[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790321392278483-9480-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 07:29:52 dist-test-slave-57fn krb5kdc[9487](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 07:29:52 dist-test-slave-57fn krb5kdc[9487](info): set up 2 sockets
Sep 25 07:29:52 dist-test-slave-57fn krb5kdc[9487](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260925 07:29:53.311842 9480 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790321392278483-9480-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790321392278483-9480-0/krb5kdc/test-user.keytab.
Sep 25 07:29:53 dist-test-slave-57fn krb5kdc[9487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321393, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T07:29:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T07:29:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.66.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.66.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.66.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab.
Entry for principal kudu/127.9.66.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab.
I20260925 07:29:53.426340 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790321392278483-9480-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790321392278483-9480-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790321392278483-9480-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.66.62:37965
--webserver_interface=127.9.66.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab
--principal=kudu/127.9.66.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:38829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.66.62:37965
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:29:53.506310 9500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:29:53.506510 9500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:29:53.506532 9500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:29:53.507728 9500 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:29:53.507771 9500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:29:53.507786 9500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:29:53.507826 9500 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:29:53.507854 9500 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:29:53.509301 9500 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:38829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790321392278483-9480-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790321392278483-9480-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.66.62:37965
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab
--principal=kudu/127.9.66.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.66.62:37965
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.66.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9500
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790321392278483-9480-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:29:53.509573 9500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:29:53.509804 9500 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 07:29:53.515223 9500 server_base.cc:1034] running on GCE node
W20260925 07:29:53.516448 9506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:29:53.516575 9505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:29:53.516871 9508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:29:53.517169 9500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:29:53.517474 9500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:29:53.518685 9500 hybrid_clock.cc:648] HybridClock initialized: now 1790321393518662 us; error 37 us; skew 500 ppm
Sep 25 07:29:53 dist-test-slave-57fn krb5kdc[9487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321393, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:29:53.527454 9500 init.cc:377] Logged in from keytab as kudu/127.9.66.62@KRBTEST.COM (short username kudu)
I20260925 07:29:53.528002 9500 webserver.cc:460] Webserver started at http://127.9.66.62:44865/ using document root <none> and password file <none>
I20260925 07:29:53.528224 9500 fs_manager.cc:359] Metadata directory not provided
I20260925 07:29:53.528268 9500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:29:53.528438 9500 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:29:53.529500 9500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790321392278483-9480-0/minicluster-data/master-0/data/instance:
uuid: "30b9c4ffa64645bfb49acf5e970f14c5"
format_stamp: "Formatted at 2026-09-25 07:29:53 on dist-test-slave-57fn"
server_key: "0f4d4e32b8fa5eff428adee19ed26ddf"
server_key_iv: "bd28feeb4541cf7a83c8e2bbf05d2882"
server_key_version: "encryptionkey@0"
I20260925 07:29:53.529942 9500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790321392278483-9480-0/minicluster-data/master-0/wal/instance:
uuid: "30b9c4ffa64645bfb49acf5e970f14c5"
format_stamp: "Formatted at 2026-09-25 07:29:53 on dist-test-slave-57fn"
server_key: "0f4d4e32b8fa5eff428adee19ed26ddf"
server_key_iv: "bd28feeb4541cf7a83c8e2bbf05d2882"
server_key_version: "encryptionkey@0"
I20260925 07:29:53.531306 9500 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 07:29:53.532033 9515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:29:53.532217 9500 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 07:29:53.532306 9500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790321392278483-9480-0/minicluster-data/master-0/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790321392278483-9480-0/minicluster-data/master-0/wal
uuid: "30b9c4ffa64645bfb49acf5e970f14c5"
format_stamp: "Formatted at 2026-09-25 07:29:53 on dist-test-slave-57fn"
server_key: "0f4d4e32b8fa5eff428adee19ed26ddf"
server_key_iv: "bd28feeb4541cf7a83c8e2bbf05d2882"
server_key_version: "encryptionkey@0"
I20260925 07:29:53.532379 9500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790321392278483-9480-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790321392278483-9480-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790321392278483-9480-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:29:53.560367 9500 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:29:53.562553 9500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:29:53.562727 9500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:29:53.567130 9500 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.62:37965
I20260925 07:29:53.567167 9567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.62:37965 every 8 connection(s)
I20260925 07:29:53.567539 9500 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
I20260925 07:29:53.568396 9568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:29:53.570714 9568 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30b9c4ffa64645bfb49acf5e970f14c5: Bootstrap starting.
I20260925 07:29:53.570811 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 9500
I20260925 07:29:53.570932 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790321392278483-9480-0/minicluster-data/master-0/wal/instance
I20260925 07:29:53.571365 9568 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 30b9c4ffa64645bfb49acf5e970f14c5: Neither blocks nor log segments found. Creating new log.
I20260925 07:29:53.571593 9480 external_mini_cluster.cc:1428] Setting key 2567641892d074d568a0f4cbb4f847f5
I20260925 07:29:53.571689 9568 log.cc:824] T 00000000000000000000000000000000 P 30b9c4ffa64645bfb49acf5e970f14c5: Log is configured to *not* fsync() on all Append() calls
I20260925 07:29:53.572459 9568 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30b9c4ffa64645bfb49acf5e970f14c5: No bootstrap required, opened a new log
I20260925 07:29:53.574190 9568 raft_consensus.cc:348] T 00000000000000000000000000000000 P 30b9c4ffa64645bfb49acf5e970f14c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30b9c4ffa64645bfb49acf5e970f14c5" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 37965 } }
I20260925 07:29:53.574352 9568 raft_consensus.cc:374] T 00000000000000000000000000000000 P 30b9c4ffa64645bfb49acf5e970f14c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:29:53.574398 9568 raft_consensus.cc:729] T 00000000000000000000000000000000 P 30b9c4ffa64645bfb49acf5e970f14c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30b9c4ffa64645bfb49acf5e970f14c5, State: Initialized, Role: FOLLOWER
I20260925 07:29:53.574525 9568 consensus_queue.cc:260] T 00000000000000000000000000000000 P 30b9c4ffa64645bfb49acf5e970f14c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30b9c4ffa64645bfb49acf5e970f14c5" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 37965 } }
I20260925 07:29:53.574621 9568 raft_consensus.cc:388] T 00000000000000000000000000000000 P 30b9c4ffa64645bfb49acf5e970f14c5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 07:29:53.574664 9568 raft_consensus.cc:482] T 00000000000000000000000000000000 P 30b9c4ffa64645bfb49acf5e970f14c5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 07:29:53.574712 9568 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 30b9c4ffa64645bfb49acf5e970f14c5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:29:53.575752 9568 raft_consensus.cc:504] T 00000000000000000000000000000000 P 30b9c4ffa64645bfb49acf5e970f14c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30b9c4ffa64645bfb49acf5e970f14c5" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 37965 } }
I20260925 07:29:53.575870 9568 leader_election.cc:302] T 00000000000000000000000000000000 P 30b9c4ffa64645bfb49acf5e970f14c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 30b9c4ffa64645bfb49acf5e970f14c5; no voters:
I20260925 07:29:53.576037 9568 leader_election.cc:288] T 00000000000000000000000000000000 P 30b9c4ffa64645bfb49acf5e970f14c5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 07:29:53.576067 9573 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 30b9c4ffa64645bfb49acf5e970f14c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:29:53.576226 9573 raft_consensus.cc:686] T 00000000000000000000000000000000 P 30b9c4ffa64645bfb49acf5e970f14c5 [term 1 LEADER]: Becoming Leader. State: Replica: 30b9c4ffa64645bfb49acf5e970f14c5, State: Running, Role: LEADER
I20260925 07:29:53.576288 9568 sys_catalog.cc:564] T 00000000000000000000000000000000 P 30b9c4ffa64645bfb49acf5e970f14c5 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:29:53.576316 9573 consensus_queue.cc:237] T 00000000000000000000000000000000 P 30b9c4ffa64645bfb49acf5e970f14c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30b9c4ffa64645bfb49acf5e970f14c5" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 37965 } }
I20260925 07:29:53.576750 9574 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30b9c4ffa64645bfb49acf5e970f14c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "30b9c4ffa64645bfb49acf5e970f14c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30b9c4ffa64645bfb49acf5e970f14c5" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 37965 } } }
I20260925 07:29:53.576835 9574 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30b9c4ffa64645bfb49acf5e970f14c5 [sys.catalog]: This master's current role is: LEADER
I20260925 07:29:53.577025 9575 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30b9c4ffa64645bfb49acf5e970f14c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 30b9c4ffa64645bfb49acf5e970f14c5. Latest consensus state: current_term: 1 leader_uuid: "30b9c4ffa64645bfb49acf5e970f14c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30b9c4ffa64645bfb49acf5e970f14c5" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 37965 } } }
I20260925 07:29:53.577111 9575 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30b9c4ffa64645bfb49acf5e970f14c5 [sys.catalog]: This master's current role is: LEADER
I20260925 07:29:53.577745 9581 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 07:29:53.578279 9581 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 07:29:53.579854 9581 catalog_manager.cc:1347] Generated new cluster ID: d003963c2cd145b1a2d95f1fe7ef5971
I20260925 07:29:53.579907 9581 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 07:29:53 dist-test-slave-57fn krb5kdc[9487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321393, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.66.62@KRBTEST.COM
I20260925 07:29:53.588142 9572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:29:53.572469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51622 (local address 127.9.66.62:37965)
0925 07:29:53.572668 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 07:29:53.572678 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 07:29:53.572951 (+ 273us) server_negotiation.cc:407] Connection header received
0925 07:29:53.574670 (+ 1719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:29:53.574676 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:29:53.574729 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:29:53.574845 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:29:53.577119 (+ 2274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:29:53.578556 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:29:53.579995 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:29:53.580404 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:29:53.580536 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:29:53.583914 (+ 3378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:29:53.583918 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:29:53.583918 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:29:53.583929 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:29:53.586216 (+ 2287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:29:53.586706 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:29:53.586709 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:29:53.586709 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:29:53.586811 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:29:53.587089 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:29:53.587089 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:29:53.587090 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:29:53.587327 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:29:53.587509 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0925 07:29:53.587768 (+ 259us) server_negotiation.cc:299] Negotiation successful
0925 07:29:53.587914 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":79,"threads_started":1}
I20260925 07:29:53.599561 9581 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 07:29:53.600071 9581 catalog_manager.cc:1504] Loading token signing keys...
I20260925 07:29:53.604605 9581 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 30b9c4ffa64645bfb49acf5e970f14c5: Generated new TSK 0
I20260925 07:29:53.604784 9581 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.66.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.66.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.66.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.1.keytab.
Entry for principal kudu/127.9.66.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.1.keytab.
I20260925 07:29:53.634305 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790321392278483-9480-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790321392278483-9480-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790321392278483-9480-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790321392278483-9480-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.66.1:0
--local_ip_for_outbound_sockets=127.9.66.1
--webserver_interface=127.9.66.1
--webserver_port=0
--tserver_master_addrs=127.9.66.62:37965
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.1.keytab
--principal=kudu/127.9.66.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:38829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:29:53.723071 9594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:29:53.723255 9594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:29:53.723278 9594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:29:53.724411 9594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:29:53.724473 9594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:29:53.724500 9594 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.66.1
I20260925 07:29:53.725979 9594 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:38829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790321392278483-9480-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790321392278483-9480-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.1.keytab
--principal=kudu/127.9.66.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.66.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790321392278483-9480-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.66.1
--webserver_port=0
--tserver_master_addrs=127.9.66.62:37965
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9594
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.66.1
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790321392278483-9480-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:29:53.726195 9594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:29:53.726438 9594 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:29:53.731343 9602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:29:53.731370 9600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:29:53.731370 9599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:29:53.731524 9594 server_base.cc:1034] running on GCE node
I20260925 07:29:53.731719 9594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:29:53.731971 9594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:29:53.733140 9594 hybrid_clock.cc:648] HybridClock initialized: now 1790321393733122 us; error 31 us; skew 500 ppm
Sep 25 07:29:53 dist-test-slave-57fn krb5kdc[9487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321393, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:29:53.735569 9594 init.cc:377] Logged in from keytab as kudu/127.9.66.1@KRBTEST.COM (short username kudu)
I20260925 07:29:53.736104 9594 webserver.cc:460] Webserver started at http://127.9.66.1:33273/ using document root <none> and password file <none>
I20260925 07:29:53.736332 9594 fs_manager.cc:359] Metadata directory not provided
I20260925 07:29:53.736398 9594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:29:53.736554 9594 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:29:53.737636 9594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790321392278483-9480-0/minicluster-data/ts-0/data/instance:
uuid: "cf29cc51cbae462eb90a894c6391fa24"
format_stamp: "Formatted at 2026-09-25 07:29:53 on dist-test-slave-57fn"
server_key: "0c67f1bbeac5c1b04a5ecf9ca6392a68"
server_key_iv: "f15a3b06e512a028f043af4b806ed2e4"
server_key_version: "encryptionkey@0"
I20260925 07:29:53.737998 9594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790321392278483-9480-0/minicluster-data/ts-0/wal/instance:
uuid: "cf29cc51cbae462eb90a894c6391fa24"
format_stamp: "Formatted at 2026-09-25 07:29:53 on dist-test-slave-57fn"
server_key: "0c67f1bbeac5c1b04a5ecf9ca6392a68"
server_key_iv: "f15a3b06e512a028f043af4b806ed2e4"
server_key_version: "encryptionkey@0"
I20260925 07:29:53.739312 9594 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:29:53.739982 9609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:29:53.740150 9594 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 07:29:53.740247 9594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790321392278483-9480-0/minicluster-data/ts-0/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790321392278483-9480-0/minicluster-data/ts-0/wal
uuid: "cf29cc51cbae462eb90a894c6391fa24"
format_stamp: "Formatted at 2026-09-25 07:29:53 on dist-test-slave-57fn"
server_key: "0c67f1bbeac5c1b04a5ecf9ca6392a68"
server_key_iv: "f15a3b06e512a028f043af4b806ed2e4"
server_key_version: "encryptionkey@0"
I20260925 07:29:53.740324 9594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790321392278483-9480-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790321392278483-9480-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790321392278483-9480-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:29:53.770919 9594 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:29:53.772887 9594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:29:53.773063 9594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:29:53.773315 9594 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:29:53.773644 9594 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:29:53.773699 9594 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:29:53.773734 9594 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:29:53.773775 9594 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:29:53.779858 9594 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.1:35891
I20260925 07:29:53.779906 9722 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.1:35891 every 8 connection(s)
I20260925 07:29:53.780251 9594 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790321392278483-9480-0/minicluster-data/ts-0/data/info.pb
Sep 25 07:29:53 dist-test-slave-57fn krb5kdc[9487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321393, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.1@KRBTEST.COM for kudu/127.9.66.62@KRBTEST.COM
I20260925 07:29:53.789330 9572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:29:53.781060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.1:35269 (local address 127.9.66.62:37965)
0925 07:29:53.781119 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 07:29:53.781122 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:29:53.781280 (+ 158us) server_negotiation.cc:407] Connection header received
0925 07:29:53.781845 (+ 565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:29:53.781847 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:29:53.781865 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:29:53.781943 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:29:53.782851 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:29:53.783401 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:29:53.784346 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:29:53.784542 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:29:53.784606 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:29:53.786633 (+ 2027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:29:53.786635 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:29:53.786635 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:29:53.786642 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:29:53.788183 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:29:53.788514 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:29:53.788514 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:29:53.788515 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:29:53.788570 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:29:53.788796 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:29:53.788797 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:29:53.788797 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:29:53.788913 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:29:53.788991 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 07:29:53.789204 (+ 213us) server_negotiation.cc:299] Negotiation successful
0925 07:29:53.789268 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 07:29:53.789391 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 9594
I20260925 07:29:53.789492 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790321392278483-9480-0/minicluster-data/ts-0/wal/instance
I20260925 07:29:53.789597 9725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:29:53.780994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:37965 (local address 127.9.66.1:35269)
0925 07:29:53.781157 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 07:29:53.781196 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 07:29:53.781720 (+ 524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:29:53.782029 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:29:53.782032 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:29:53.782560 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:29:53.782742 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:29:53.782746 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:29:53.783484 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:29:53.783484 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:29:53.784256 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:29:53.784258 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:29:53.784620 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:29:53.784621 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:29:53.784715 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:29:53.785151 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:29:53.785159 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:29:53.786554 (+ 1395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:29:53.788255 (+ 1701us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:29:53.788260 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:29:53.788261 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:29:53.788447 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:29:53.788631 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:29:53.788631 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:29:53.788632 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:29:53.788718 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:29:53.788992 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:29:53.788996 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:29:53.789098 (+ 102us) client_negotiation.cc:769] Sending connection context
0925 07:29:53.789173 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 07:29:53.789343 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":89,"threads_started":1}
I20260925 07:29:53.789677 9480 external_mini_cluster.cc:1428] Setting key 264ddb91c0efeb9a6074e5b68c130042
I20260925 07:29:53.790131 9723 heartbeater.cc:347] Connected to a master server at 127.9.66.62:37965
I20260925 07:29:53.790241 9723 heartbeater.cc:464] Registering TS with master...
I20260925 07:29:53.790428 9723 heartbeater.cc:511] Master 127.9.66.62:37965 requested a full tablet report, sending...
I20260925 07:29:53.790930 9532 ts_manager.cc:194] Registered new tserver with Master: cf29cc51cbae462eb90a894c6391fa24 (127.9.66.1:35891)
I20260925 07:29:53.791517 9532 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.66.1@KRBTEST.COM'} at 127.9.66.1:35269
WARNING: no policy specified for kudu/127.9.66.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.66.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.66.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.2.keytab.
Entry for principal kudu/127.9.66.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.2.keytab.
I20260925 07:29:53.812794 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790321392278483-9480-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790321392278483-9480-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790321392278483-9480-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790321392278483-9480-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.66.2:0
--local_ip_for_outbound_sockets=127.9.66.2
--webserver_interface=127.9.66.2
--webserver_port=0
--tserver_master_addrs=127.9.66.62:37965
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.2.keytab
--principal=kudu/127.9.66.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:38829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:29:53.891536 9728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:29:53.891738 9728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:29:53.891762 9728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:29:53.892838 9728 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:29:53.892869 9728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:29:53.892894 9728 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.66.2
I20260925 07:29:53.894253 9728 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:38829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790321392278483-9480-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790321392278483-9480-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.2.keytab
--principal=kudu/127.9.66.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.66.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790321392278483-9480-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.66.2
--webserver_port=0
--tserver_master_addrs=127.9.66.62:37965
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9728
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.66.2
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790321392278483-9480-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:29:53.894423 9728 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:29:53.894577 9728 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:29:53.899425 9736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:29:53.899468 9734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:29:53.899437 9733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:29:53.899523 9728 server_base.cc:1034] running on GCE node
I20260925 07:29:53.899839 9728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:29:53.900027 9728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:29:53.901153 9728 hybrid_clock.cc:648] HybridClock initialized: now 1790321393901139 us; error 38 us; skew 500 ppm
Sep 25 07:29:53 dist-test-slave-57fn krb5kdc[9487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321393, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:29:53.903537 9728 init.cc:377] Logged in from keytab as kudu/127.9.66.2@KRBTEST.COM (short username kudu)
I20260925 07:29:53.904029 9728 webserver.cc:460] Webserver started at http://127.9.66.2:34237/ using document root <none> and password file <none>
I20260925 07:29:53.904244 9728 fs_manager.cc:359] Metadata directory not provided
I20260925 07:29:53.904309 9728 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:29:53.904448 9728 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:29:53.905393 9728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790321392278483-9480-0/minicluster-data/ts-1/data/instance:
uuid: "b8c4330ae54e4be78089aaa4358b9d8e"
format_stamp: "Formatted at 2026-09-25 07:29:53 on dist-test-slave-57fn"
server_key: "2cca691abc82fc925a36bea2fd3cbf65"
server_key_iv: "9c5c484cbc6ab58afc890f7b0d4abf22"
server_key_version: "encryptionkey@0"
I20260925 07:29:53.905766 9728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790321392278483-9480-0/minicluster-data/ts-1/wal/instance:
uuid: "b8c4330ae54e4be78089aaa4358b9d8e"
format_stamp: "Formatted at 2026-09-25 07:29:53 on dist-test-slave-57fn"
server_key: "2cca691abc82fc925a36bea2fd3cbf65"
server_key_iv: "9c5c484cbc6ab58afc890f7b0d4abf22"
server_key_version: "encryptionkey@0"
I20260925 07:29:53.906997 9728 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:29:53.907687 9743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:29:53.907884 9728 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 07:29:53.907980 9728 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790321392278483-9480-0/minicluster-data/ts-1/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790321392278483-9480-0/minicluster-data/ts-1/wal
uuid: "b8c4330ae54e4be78089aaa4358b9d8e"
format_stamp: "Formatted at 2026-09-25 07:29:53 on dist-test-slave-57fn"
server_key: "2cca691abc82fc925a36bea2fd3cbf65"
server_key_iv: "9c5c484cbc6ab58afc890f7b0d4abf22"
server_key_version: "encryptionkey@0"
I20260925 07:29:53.908064 9728 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790321392278483-9480-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790321392278483-9480-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790321392278483-9480-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:29:53.936980 9728 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:29:53.939688 9728 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:29:53.939846 9728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:29:53.940080 9728 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:29:53.940376 9728 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:29:53.940438 9728 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:29:53.940478 9728 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:29:53.940507 9728 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:29:53.946502 9728 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.2:41841
I20260925 07:29:53.946549 9856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.2:41841 every 8 connection(s)
I20260925 07:29:53.946837 9728 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790321392278483-9480-0/minicluster-data/ts-1/data/info.pb
I20260925 07:29:53.947279 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 9728
I20260925 07:29:53.947400 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790321392278483-9480-0/minicluster-data/ts-1/wal/instance
I20260925 07:29:53.947587 9480 external_mini_cluster.cc:1428] Setting key 06e0433096a8d6b8701c9488d716954f
Sep 25 07:29:53 dist-test-slave-57fn krb5kdc[9487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321393, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.2@KRBTEST.COM for kudu/127.9.66.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.66.3@KRBTEST.COM; defaulting to no policy
I20260925 07:29:53.956900 9572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:29:53.948136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.2:52111 (local address 127.9.66.62:37965)
0925 07:29:53.948176 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 07:29:53.948178 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:29:53.948538 (+ 360us) server_negotiation.cc:407] Connection header received
0925 07:29:53.948965 (+ 427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:29:53.948966 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:29:53.948979 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:29:53.949044 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:29:53.950047 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:29:53.950724 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:29:53.951652 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:29:53.951852 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:29:53.951947 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:29:53.954366 (+ 2419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:29:53.954367 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:29:53.954367 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:29:53.954374 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:29:53.955685 (+ 1311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:29:53.956175 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:29:53.956176 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:29:53.956176 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:29:53.956226 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:29:53.956433 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:29:53.956433 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:29:53.956433 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:29:53.956521 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:29:53.956565 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0925 07:29:53.956758 (+ 193us) server_negotiation.cc:299] Negotiation successful
0925 07:29:53.956815 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 07:29:53.957185 9859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:29:53.948148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:37965 (local address 127.9.66.2:52111)
0925 07:29:53.948427 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0925 07:29:53.948461 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 07:29:53.948883 (+ 422us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:29:53.949109 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:29:53.949113 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:29:53.949767 (+ 654us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:29:53.949954 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:29:53.949958 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:29:53.950804 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:29:53.950804 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:29:53.951559 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:29:53.951561 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:29:53.951954 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:29:53.951955 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:29:53.952087 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:29:53.952621 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:29:53.952628 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:29:53.954279 (+ 1651us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:29:53.955788 (+ 1509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:29:53.955794 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:29:53.955796 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:29:53.956099 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:29:53.956288 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:29:53.956288 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:29:53.956289 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:29:53.956379 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:29:53.956571 (+ 192us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:29:53.956574 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:29:53.956675 (+ 101us) client_negotiation.cc:769] Sending connection context
0925 07:29:53.956749 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 07:29:53.956941 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":86,"threads_started":1}
I20260925 07:29:53.957686 9857 heartbeater.cc:347] Connected to a master server at 127.9.66.62:37965
I20260925 07:29:53.957760 9857 heartbeater.cc:464] Registering TS with master...
I20260925 07:29:53.957890 9857 heartbeater.cc:511] Master 127.9.66.62:37965 requested a full tablet report, sending...
I20260925 07:29:53.958258 9532 ts_manager.cc:194] Registered new tserver with Master: b8c4330ae54e4be78089aaa4358b9d8e (127.9.66.2:41841)
I20260925 07:29:53.958655 9532 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.66.2@KRBTEST.COM'} at 127.9.66.2:52111
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.66.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.66.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.3.keytab.
Entry for principal kudu/127.9.66.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.3.keytab.
I20260925 07:29:53.972091 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790321392278483-9480-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790321392278483-9480-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790321392278483-9480-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790321392278483-9480-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.66.3:0
--local_ip_for_outbound_sockets=127.9.66.3
--webserver_interface=127.9.66.3
--webserver_port=0
--tserver_master_addrs=127.9.66.62:37965
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.3.keytab
--principal=kudu/127.9.66.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:38829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:29:54.050931 9862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:29:54.051110 9862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:29:54.051133 9862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:29:54.052222 9862 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:29:54.052255 9862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:29:54.052280 9862 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.66.3
I20260925 07:29:54.053710 9862 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:38829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790321392278483-9480-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790321392278483-9480-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.3.keytab
--principal=kudu/127.9.66.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.66.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790321392278483-9480-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.66.3
--webserver_port=0
--tserver_master_addrs=127.9.66.62:37965
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9862
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.66.3
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790321392278483-9480-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:29:54.053871 9862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:29:54.054032 9862 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:29:54.058724 9867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:29:54.058751 9870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:29:54.058784 9862 server_base.cc:1034] running on GCE node
W20260925 07:29:54.058768 9868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:29:54.059201 9862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:29:54.059450 9862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:29:54.060603 9862 hybrid_clock.cc:648] HybridClock initialized: now 1790321394060592 us; error 28 us; skew 500 ppm
Sep 25 07:29:54 dist-test-slave-57fn krb5kdc[9487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321394, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:29:54.062978 9862 init.cc:377] Logged in from keytab as kudu/127.9.66.3@KRBTEST.COM (short username kudu)
I20260925 07:29:54.063436 9862 webserver.cc:460] Webserver started at http://127.9.66.3:45881/ using document root <none> and password file <none>
I20260925 07:29:54.063665 9862 fs_manager.cc:359] Metadata directory not provided
I20260925 07:29:54.063711 9862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:29:54.063833 9862 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:29:54.064819 9862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790321392278483-9480-0/minicluster-data/ts-2/data/instance:
uuid: "ce40990652f54761abcf4276804f9355"
format_stamp: "Formatted at 2026-09-25 07:29:54 on dist-test-slave-57fn"
server_key: "49e629676c65be2e7ff5e799e85185db"
server_key_iv: "e64aa810f9288b33f00bb63ee9144eb7"
server_key_version: "encryptionkey@0"
I20260925 07:29:54.065171 9862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790321392278483-9480-0/minicluster-data/ts-2/wal/instance:
uuid: "ce40990652f54761abcf4276804f9355"
format_stamp: "Formatted at 2026-09-25 07:29:54 on dist-test-slave-57fn"
server_key: "49e629676c65be2e7ff5e799e85185db"
server_key_iv: "e64aa810f9288b33f00bb63ee9144eb7"
server_key_version: "encryptionkey@0"
I20260925 07:29:54.066500 9862 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 07:29:54.067195 9877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:29:54.067360 9862 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 07:29:54.067456 9862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790321392278483-9480-0/minicluster-data/ts-2/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790321392278483-9480-0/minicluster-data/ts-2/wal
uuid: "ce40990652f54761abcf4276804f9355"
format_stamp: "Formatted at 2026-09-25 07:29:54 on dist-test-slave-57fn"
server_key: "49e629676c65be2e7ff5e799e85185db"
server_key_iv: "e64aa810f9288b33f00bb63ee9144eb7"
server_key_version: "encryptionkey@0"
I20260925 07:29:54.067535 9862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790321392278483-9480-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790321392278483-9480-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790321392278483-9480-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:29:54.081930 9862 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:29:54.083616 9862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:29:54.083774 9862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:29:54.083994 9862 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:29:54.084272 9862 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:29:54.084324 9862 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:29:54.084363 9862 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:29:54.084393 9862 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:29:54.090444 9862 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.3:45103
I20260925 07:29:54.090550 9990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.3:45103 every 8 connection(s)
I20260925 07:29:54.090806 9862 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790321392278483-9480-0/minicluster-data/ts-2/data/info.pb
I20260925 07:29:54.096180 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 9862
I20260925 07:29:54.096262 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790321392278483-9480-0/minicluster-data/ts-2/wal/instance
I20260925 07:29:54.096426 9480 external_mini_cluster.cc:1428] Setting key 63cc034d464f940455dfcdb3c27baff1
Sep 25 07:29:54 dist-test-slave-57fn krb5kdc[9487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321394, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.3@KRBTEST.COM for kudu/127.9.66.62@KRBTEST.COM
I20260925 07:29:54.099932 9572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:29:54.091563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.3:44649 (local address 127.9.66.62:37965)
0925 07:29:54.091623 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 07:29:54.091625 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:29:54.091820 (+ 195us) server_negotiation.cc:407] Connection header received
0925 07:29:54.092272 (+ 452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:29:54.092272 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:29:54.092281 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:29:54.092334 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:29:54.093209 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:29:54.093779 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:29:54.094691 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:29:54.094861 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:29:54.094908 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:29:54.097225 (+ 2317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:29:54.097226 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:29:54.097227 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:29:54.097233 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:29:54.098749 (+ 1516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:29:54.099089 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:29:54.099090 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:29:54.099090 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:29:54.099141 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:29:54.099411 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:29:54.099411 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:29:54.099412 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:29:54.099506 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:29:54.099553 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 07:29:54.099795 (+ 242us) server_negotiation.cc:299] Negotiation successful
0925 07:29:54.099861 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 07:29:54.100081 9993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:29:54.091551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:37965 (local address 127.9.66.3:44649)
0925 07:29:54.091701 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 07:29:54.091738 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 07:29:54.092193 (+ 455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:29:54.092405 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:29:54.092409 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:29:54.092958 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:29:54.093137 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:29:54.093141 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:29:54.093862 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:29:54.093863 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:29:54.094619 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:29:54.094621 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:29:54.094929 (+ 308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:29:54.094929 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:29:54.095021 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:29:54.095422 (+ 401us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:29:54.095430 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:29:54.097148 (+ 1718us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:29:54.098818 (+ 1670us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:29:54.098822 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:29:54.098824 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:29:54.099025 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:29:54.099223 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:29:54.099224 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:29:54.099224 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:29:54.099344 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:29:54.099581 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:29:54.099585 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:29:54.099689 (+ 104us) client_negotiation.cc:769] Sending connection context
0925 07:29:54.099770 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 07:29:54.099881 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":81,"threads_started":1}
I20260925 07:29:54.100508 9991 heartbeater.cc:347] Connected to a master server at 127.9.66.62:37965
I20260925 07:29:54.100605 9991 heartbeater.cc:464] Registering TS with master...
I20260925 07:29:54.100769 9991 heartbeater.cc:511] Master 127.9.66.62:37965 requested a full tablet report, sending...
I20260925 07:29:54.101130 9532 ts_manager.cc:194] Registered new tserver with Master: ce40990652f54761abcf4276804f9355 (127.9.66.3:45103)
I20260925 07:29:54.101473 9532 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.66.3@KRBTEST.COM'} at 127.9.66.3:44649
I20260925 07:29:54.108389 9480 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 07:29:54 dist-test-slave-57fn krb5kdc[9487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321394, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 07:29:54 dist-test-slave-57fn krb5kdc[9487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321394, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.9.66.1@KRBTEST.COM
I20260925 07:29:54.130926 9998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:29:54.123466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52902 (local address 127.9.66.1:35891)
0925 07:29:54.123615 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 07:29:54.123620 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 07:29:54.123628 (+ 8us) server_negotiation.cc:407] Connection header received
0925 07:29:54.123651 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:29:54.123653 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:29:54.123665 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:29:54.123747 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:29:54.124298 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:29:54.125011 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:29:54.125794 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:29:54.126092 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:29:54.126171 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:29:54.128294 (+ 2123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:29:54.128301 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:29:54.128302 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:29:54.128308 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:29:54.129744 (+ 1436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:29:54.130097 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:29:54.130102 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:29:54.130102 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:29:54.130171 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:29:54.130378 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:29:54.130379 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:29:54.130379 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:29:54.130504 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:29:54.130658 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0925 07:29:54.130728 (+ 70us) server_negotiation.cc:299] Negotiation successful
0925 07:29:54.130801 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":63,"threads_started":1}
W20260925 07:29:54.131284 9637 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52902
Sep 25 07:29:54 dist-test-slave-57fn krb5kdc[9487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321394, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 07:29:54 dist-test-slave-57fn krb5kdc[9487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321394, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.66.1@KRBTEST.COM
I20260925 07:29:54.151619 9998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:29:54.145352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52908 (local address 127.9.66.1:35891)
0925 07:29:54.145410 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 07:29:54.145412 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:29:54.145500 (+ 88us) server_negotiation.cc:407] Connection header received
0925 07:29:54.145597 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:29:54.145598 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:29:54.145607 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:29:54.145653 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:29:54.146148 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:29:54.146588 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:29:54.147244 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:29:54.147438 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:29:54.147486 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:29:54.149587 (+ 2101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:29:54.149590 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:29:54.149591 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:29:54.149601 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:29:54.150717 (+ 1116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:29:54.151007 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:29:54.151008 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:29:54.151008 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:29:54.151053 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:29:54.151214 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:29:54.151215 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:29:54.151215 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:29:54.151288 (+ 73us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:29:54.151367 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 07:29:54.151468 (+ 101us) server_negotiation.cc:299] Negotiation successful
0925 07:29:54.151522 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 07:29:54.152308 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 9594
I20260925 07:29:54.156683 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 9728
I20260925 07:29:54.161093 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 9862
I20260925 07:29:54.165645 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 9500
2026-09-25T07:29:54Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1894 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790321392278483-9480-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 07:29:54 dist-test-slave-57fn krb5kdc[10005](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 07:29:54 dist-test-slave-57fn krb5kdc[10005](info): set up 2 sockets
Sep 25 07:29:54 dist-test-slave-57fn krb5kdc[10005](info): commencing operation
krb5kdc: starting...
W20260925 07:29:55.197069 9480 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790321392278483-9480-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790321392278483-9480-0/krb5kdc/test-user.keytab.
Sep 25 07:29:55 dist-test-slave-57fn krb5kdc[10005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321395, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T07:29:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T07:29:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.66.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.66.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.66.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab.
Entry for principal kudu/127.9.66.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab.
I20260925 07:29:55.308071 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790321392278483-9480-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790321392278483-9480-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790321392278483-9480-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.66.62:38333
--webserver_interface=127.9.66.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab
--principal=kudu/127.9.66.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:40131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.66.62:38333
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:29:55.385210 10018 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:29:55.385433 10018 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:29:55.385469 10018 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:29:55.386624 10018 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:29:55.386690 10018 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:29:55.386715 10018 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:29:55.386746 10018 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:29:55.386770 10018 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:29:55.388208 10018 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:40131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790321392278483-9480-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790321392278483-9480-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.66.62:38333
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab
--principal=kudu/127.9.66.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.66.62:38333
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.66.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10018
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790321392278483-9480-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:29:55.388468 10018 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:29:55.388681 10018 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:29:55.393415 10024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:29:55.393471 10023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:29:55.393482 10026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:29:55.393643 10018 server_base.cc:1034] running on GCE node
I20260925 07:29:55.393879 10018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:29:55.394155 10018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:29:55.395371 10018 hybrid_clock.cc:648] HybridClock initialized: now 1790321395395336 us; error 37 us; skew 500 ppm
Sep 25 07:29:55 dist-test-slave-57fn krb5kdc[10005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321395, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:29:55.397861 10018 init.cc:377] Logged in from keytab as kudu/127.9.66.62@KRBTEST.COM (short username kudu)
I20260925 07:29:55.398268 10018 webserver.cc:460] Webserver started at http://127.9.66.62:44621/ using document root <none> and password file <none>
I20260925 07:29:55.398509 10018 fs_manager.cc:359] Metadata directory not provided
I20260925 07:29:55.398577 10018 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:29:55.398729 10018 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:29:55.399778 10018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790321392278483-9480-0/minicluster-data/master-0/data/instance:
uuid: "cec546e836aa4293be7b74f42767a63e"
format_stamp: "Formatted at 2026-09-25 07:29:55 on dist-test-slave-57fn"
server_key: "468a16a87a18843bb55de43eca5743f1"
server_key_iv: "477a57f0df79d79758fbb36a399a769a"
server_key_version: "encryptionkey@0"
I20260925 07:29:55.400173 10018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790321392278483-9480-0/minicluster-data/master-0/wal/instance:
uuid: "cec546e836aa4293be7b74f42767a63e"
format_stamp: "Formatted at 2026-09-25 07:29:55 on dist-test-slave-57fn"
server_key: "468a16a87a18843bb55de43eca5743f1"
server_key_iv: "477a57f0df79d79758fbb36a399a769a"
server_key_version: "encryptionkey@0"
I20260925 07:29:55.401624 10018 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260925 07:29:55.402397 10033 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:29:55.402562 10018 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 07:29:55.402678 10018 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790321392278483-9480-0/minicluster-data/master-0/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790321392278483-9480-0/minicluster-data/master-0/wal
uuid: "cec546e836aa4293be7b74f42767a63e"
format_stamp: "Formatted at 2026-09-25 07:29:55 on dist-test-slave-57fn"
server_key: "468a16a87a18843bb55de43eca5743f1"
server_key_iv: "477a57f0df79d79758fbb36a399a769a"
server_key_version: "encryptionkey@0"
I20260925 07:29:55.402765 10018 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790321392278483-9480-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790321392278483-9480-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790321392278483-9480-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:29:55.421082 10018 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:29:55.422963 10018 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:29:55.423126 10018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:29:55.426860 10018 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.62:38333
I20260925 07:29:55.426895 10085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.62:38333 every 8 connection(s)
I20260925 07:29:55.427232 10018 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
I20260925 07:29:55.428056 10086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:29:55.430220 10086 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cec546e836aa4293be7b74f42767a63e: Bootstrap starting.
I20260925 07:29:55.430770 10086 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cec546e836aa4293be7b74f42767a63e: Neither blocks nor log segments found. Creating new log.
I20260925 07:29:55.431049 10086 log.cc:824] T 00000000000000000000000000000000 P cec546e836aa4293be7b74f42767a63e: Log is configured to *not* fsync() on all Append() calls
I20260925 07:29:55.431689 10086 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cec546e836aa4293be7b74f42767a63e: No bootstrap required, opened a new log
I20260925 07:29:55.432096 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 10018
I20260925 07:29:55.432183 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790321392278483-9480-0/minicluster-data/master-0/wal/instance
I20260925 07:29:55.432343 9480 external_mini_cluster.cc:1428] Setting key 6ca03c825032ae119f77ce14e07d69db
I20260925 07:29:55.434181 10086 raft_consensus.cc:348] T 00000000000000000000000000000000 P cec546e836aa4293be7b74f42767a63e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cec546e836aa4293be7b74f42767a63e" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 38333 } }
I20260925 07:29:55.434373 10086 raft_consensus.cc:374] T 00000000000000000000000000000000 P cec546e836aa4293be7b74f42767a63e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:29:55.434412 10086 raft_consensus.cc:729] T 00000000000000000000000000000000 P cec546e836aa4293be7b74f42767a63e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cec546e836aa4293be7b74f42767a63e, State: Initialized, Role: FOLLOWER
I20260925 07:29:55.434542 10086 consensus_queue.cc:260] T 00000000000000000000000000000000 P cec546e836aa4293be7b74f42767a63e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cec546e836aa4293be7b74f42767a63e" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 38333 } }
I20260925 07:29:55.434641 10086 raft_consensus.cc:388] T 00000000000000000000000000000000 P cec546e836aa4293be7b74f42767a63e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 07:29:55.434695 10086 raft_consensus.cc:482] T 00000000000000000000000000000000 P cec546e836aa4293be7b74f42767a63e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 07:29:55.434746 10086 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cec546e836aa4293be7b74f42767a63e [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:29:55.435266 10086 raft_consensus.cc:504] T 00000000000000000000000000000000 P cec546e836aa4293be7b74f42767a63e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cec546e836aa4293be7b74f42767a63e" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 38333 } }
I20260925 07:29:55.435357 10086 leader_election.cc:302] T 00000000000000000000000000000000 P cec546e836aa4293be7b74f42767a63e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cec546e836aa4293be7b74f42767a63e; no voters:
I20260925 07:29:55.435546 10086 leader_election.cc:288] T 00000000000000000000000000000000 P cec546e836aa4293be7b74f42767a63e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 07:29:55.435568 10091 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cec546e836aa4293be7b74f42767a63e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:29:55.435696 10091 raft_consensus.cc:686] T 00000000000000000000000000000000 P cec546e836aa4293be7b74f42767a63e [term 1 LEADER]: Becoming Leader. State: Replica: cec546e836aa4293be7b74f42767a63e, State: Running, Role: LEADER
I20260925 07:29:55.435772 10091 consensus_queue.cc:237] T 00000000000000000000000000000000 P cec546e836aa4293be7b74f42767a63e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cec546e836aa4293be7b74f42767a63e" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 38333 } }
I20260925 07:29:55.435809 10086 sys_catalog.cc:564] T 00000000000000000000000000000000 P cec546e836aa4293be7b74f42767a63e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:29:55.436066 10091 sys_catalog.cc:455] T 00000000000000000000000000000000 P cec546e836aa4293be7b74f42767a63e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cec546e836aa4293be7b74f42767a63e. Latest consensus state: current_term: 1 leader_uuid: "cec546e836aa4293be7b74f42767a63e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cec546e836aa4293be7b74f42767a63e" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 38333 } } }
I20260925 07:29:55.436133 10091 sys_catalog.cc:458] T 00000000000000000000000000000000 P cec546e836aa4293be7b74f42767a63e [sys.catalog]: This master's current role is: LEADER
I20260925 07:29:55.436949 10093 sys_catalog.cc:455] T 00000000000000000000000000000000 P cec546e836aa4293be7b74f42767a63e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cec546e836aa4293be7b74f42767a63e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cec546e836aa4293be7b74f42767a63e" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 38333 } } }
I20260925 07:29:55.437033 10093 sys_catalog.cc:458] T 00000000000000000000000000000000 P cec546e836aa4293be7b74f42767a63e [sys.catalog]: This master's current role is: LEADER
W20260925 07:29:55.437552 10106 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cec546e836aa4293be7b74f42767a63e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 07:29:55.437603 10106 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 07:29:55.438176 10096 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 07:29:55.438393 10096 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 25 07:29:55 dist-test-slave-57fn krb5kdc[10005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321395, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.66.62@KRBTEST.COM
I20260925 07:29:55.440414 10096 catalog_manager.cc:1347] Generated new cluster ID: cbd481038585404397994ee0b8bd4e09
I20260925 07:29:55.440486 10096 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 07:29:55.444100 10090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:29:55.432850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57804 (local address 127.9.66.62:38333)
0925 07:29:55.433037 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0925 07:29:55.433044 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 07:29:55.433067 (+ 23us) server_negotiation.cc:407] Connection header received
0925 07:29:55.433590 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:29:55.433596 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:29:55.433669 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:29:55.433775 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:29:55.434605 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:29:55.435996 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:29:55.436765 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:29:55.437419 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:29:55.437537 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:29:55.440397 (+ 2860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:29:55.440401 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:29:55.440401 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:29:55.440412 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:29:55.442420 (+ 2008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:29:55.442842 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:29:55.442845 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:29:55.442845 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:29:55.442962 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:29:55.443166 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:29:55.443167 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:29:55.443168 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:29:55.443456 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:29:55.443620 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0925 07:29:55.443711 (+ 91us) server_negotiation.cc:299] Negotiation successful
0925 07:29:55.443874 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":96,"threads_started":1}
I20260925 07:29:55.458560 10096 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 07:29:55.459221 10096 catalog_manager.cc:1504] Loading token signing keys...
I20260925 07:29:55.467661 10096 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cec546e836aa4293be7b74f42767a63e: Generated new TSK 0
I20260925 07:29:55.467819 10096 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.66.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.66.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.66.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.1.keytab.
Entry for principal kudu/127.9.66.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.1.keytab.
I20260925 07:29:55.496773 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790321392278483-9480-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790321392278483-9480-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790321392278483-9480-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790321392278483-9480-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.66.1:0
--local_ip_for_outbound_sockets=127.9.66.1
--webserver_interface=127.9.66.1
--webserver_port=0
--tserver_master_addrs=127.9.66.62:38333
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.1.keytab
--principal=kudu/127.9.66.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:40131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:29:55.576963 10112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:29:55.577176 10112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:29:55.577199 10112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:29:55.578354 10112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:29:55.578393 10112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:29:55.578419 10112 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.66.1
I20260925 07:29:55.579875 10112 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:40131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790321392278483-9480-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790321392278483-9480-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.1.keytab
--principal=kudu/127.9.66.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.66.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790321392278483-9480-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.66.1
--webserver_port=0
--tserver_master_addrs=127.9.66.62:38333
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10112
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.66.1
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790321392278483-9480-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:29:55.580102 10112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:29:55.580328 10112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:29:55.585173 10120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:29:55.585237 10118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:29:55.585215 10117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:29:55.585387 10112 server_base.cc:1034] running on GCE node
I20260925 07:29:55.585623 10112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:29:55.585804 10112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:29:55.586934 10112 hybrid_clock.cc:648] HybridClock initialized: now 1790321395586902 us; error 49 us; skew 500 ppm
Sep 25 07:29:55 dist-test-slave-57fn krb5kdc[10005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321395, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:29:55.589385 10112 init.cc:377] Logged in from keytab as kudu/127.9.66.1@KRBTEST.COM (short username kudu)
I20260925 07:29:55.589876 10112 webserver.cc:460] Webserver started at http://127.9.66.1:45321/ using document root <none> and password file <none>
I20260925 07:29:55.590106 10112 fs_manager.cc:359] Metadata directory not provided
I20260925 07:29:55.590178 10112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:29:55.590341 10112 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:29:55.591426 10112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790321392278483-9480-0/minicluster-data/ts-0/data/instance:
uuid: "4ae4d23c4515478ca0b3f2b60c9ca78e"
format_stamp: "Formatted at 2026-09-25 07:29:55 on dist-test-slave-57fn"
server_key: "4bd8a8171390eb802b62f30d2aeab65e"
server_key_iv: "5bc96cc9f1462c373d259c7eb995daa2"
server_key_version: "encryptionkey@0"
I20260925 07:29:55.591773 10112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790321392278483-9480-0/minicluster-data/ts-0/wal/instance:
uuid: "4ae4d23c4515478ca0b3f2b60c9ca78e"
format_stamp: "Formatted at 2026-09-25 07:29:55 on dist-test-slave-57fn"
server_key: "4bd8a8171390eb802b62f30d2aeab65e"
server_key_iv: "5bc96cc9f1462c373d259c7eb995daa2"
server_key_version: "encryptionkey@0"
I20260925 07:29:55.593081 10112 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 07:29:55.593959 10127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:29:55.594130 10112 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 07:29:55.594240 10112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790321392278483-9480-0/minicluster-data/ts-0/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790321392278483-9480-0/minicluster-data/ts-0/wal
uuid: "4ae4d23c4515478ca0b3f2b60c9ca78e"
format_stamp: "Formatted at 2026-09-25 07:29:55 on dist-test-slave-57fn"
server_key: "4bd8a8171390eb802b62f30d2aeab65e"
server_key_iv: "5bc96cc9f1462c373d259c7eb995daa2"
server_key_version: "encryptionkey@0"
I20260925 07:29:55.594327 10112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790321392278483-9480-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790321392278483-9480-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790321392278483-9480-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:29:55.606882 10112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:29:55.608644 10112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:29:55.608834 10112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:29:55.609045 10112 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:29:55.609349 10112 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:29:55.609400 10112 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:29:55.609434 10112 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:29:55.609463 10112 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:29:55.615657 10112 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.1:42783
I20260925 07:29:55.615694 10240 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.1:42783 every 8 connection(s)
I20260925 07:29:55.616008 10112 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790321392278483-9480-0/minicluster-data/ts-0/data/info.pb
I20260925 07:29:55.621098 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 10112
I20260925 07:29:55.621199 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790321392278483-9480-0/minicluster-data/ts-0/wal/instance
I20260925 07:29:55.621364 9480 external_mini_cluster.cc:1428] Setting key 61f2823d39bac1aa0148d92700c09c74
Sep 25 07:29:55 dist-test-slave-57fn krb5kdc[10005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321395, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.1@KRBTEST.COM for kudu/127.9.66.62@KRBTEST.COM
I20260925 07:29:55.626324 10090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:29:55.616834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.1:43223 (local address 127.9.66.62:38333)
0925 07:29:55.616923 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0925 07:29:55.616925 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:29:55.617100 (+ 175us) server_negotiation.cc:407] Connection header received
0925 07:29:55.618131 (+ 1031us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:29:55.618133 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:29:55.618147 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:29:55.618198 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:29:55.618794 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:29:55.619303 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:29:55.620358 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:29:55.620578 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:29:55.620669 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:29:55.623061 (+ 2392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:29:55.623062 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:29:55.623063 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:29:55.623070 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:29:55.624839 (+ 1769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:29:55.625277 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:29:55.625278 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:29:55.625279 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:29:55.625344 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:29:55.625650 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:29:55.625651 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:29:55.625652 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:29:55.625765 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:29:55.625886 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0925 07:29:55.626151 (+ 265us) server_negotiation.cc:299] Negotiation successful
0925 07:29:55.626237 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 07:29:55.626498 10243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:29:55.616751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:38333 (local address 127.9.66.1:43223)
0925 07:29:55.616973 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0925 07:29:55.617017 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 07:29:55.618018 (+ 1001us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:29:55.618282 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:29:55.618286 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:29:55.618524 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:29:55.618693 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:29:55.618697 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:29:55.619432 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:29:55.619433 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:29:55.620240 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:29:55.620242 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:29:55.620679 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:29:55.620680 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:29:55.620807 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:29:55.621360 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:29:55.621368 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:29:55.622944 (+ 1576us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:29:55.624953 (+ 2009us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:29:55.624960 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:29:55.624962 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:29:55.625186 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:29:55.625412 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:29:55.625413 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:29:55.625413 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:29:55.625581 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:29:55.625879 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:29:55.625885 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:29:55.626019 (+ 134us) client_negotiation.cc:769] Sending connection context
0925 07:29:55.626118 (+ 99us) client_negotiation.cc:240] Negotiation successful
0925 07:29:55.626264 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":82,"threads_started":1}
I20260925 07:29:55.626958 10241 heartbeater.cc:347] Connected to a master server at 127.9.66.62:38333
I20260925 07:29:55.627072 10241 heartbeater.cc:464] Registering TS with master...
I20260925 07:29:55.627264 10241 heartbeater.cc:511] Master 127.9.66.62:38333 requested a full tablet report, sending...
I20260925 07:29:55.627749 10050 ts_manager.cc:194] Registered new tserver with Master: 4ae4d23c4515478ca0b3f2b60c9ca78e (127.9.66.1:42783)
I20260925 07:29:55.628345 10050 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.66.1@KRBTEST.COM'} at 127.9.66.1:43223
WARNING: no policy specified for kudu/127.9.66.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.66.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.66.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.2.keytab.
Entry for principal kudu/127.9.66.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.2.keytab.
I20260925 07:29:55.646214 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790321392278483-9480-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790321392278483-9480-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790321392278483-9480-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790321392278483-9480-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.66.2:0
--local_ip_for_outbound_sockets=127.9.66.2
--webserver_interface=127.9.66.2
--webserver_port=0
--tserver_master_addrs=127.9.66.62:38333
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.2.keytab
--principal=kudu/127.9.66.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:40131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:29:55.723891 10246 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:29:55.724063 10246 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:29:55.724085 10246 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:29:55.725101 10246 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:29:55.725132 10246 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:29:55.725157 10246 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.66.2
I20260925 07:29:55.726490 10246 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:40131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790321392278483-9480-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790321392278483-9480-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.2.keytab
--principal=kudu/127.9.66.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.66.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790321392278483-9480-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.66.2
--webserver_port=0
--tserver_master_addrs=127.9.66.62:38333
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10246
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.66.2
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790321392278483-9480-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:29:55.726667 10246 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:29:55.726815 10246 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:29:55.731624 10251 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:29:55.731619 10252 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:29:55.731750 10246 server_base.cc:1034] running on GCE node
W20260925 07:29:55.731606 10254 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:29:55.732012 10246 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:29:55.732189 10246 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:29:55.733351 10246 hybrid_clock.cc:648] HybridClock initialized: now 1790321395733336 us; error 32 us; skew 500 ppm
Sep 25 07:29:55 dist-test-slave-57fn krb5kdc[10005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321395, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:29:55.735877 10246 init.cc:377] Logged in from keytab as kudu/127.9.66.2@KRBTEST.COM (short username kudu)
I20260925 07:29:55.736447 10246 webserver.cc:460] Webserver started at http://127.9.66.2:33689/ using document root <none> and password file <none>
I20260925 07:29:55.736692 10246 fs_manager.cc:359] Metadata directory not provided
I20260925 07:29:55.736768 10246 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:29:55.736917 10246 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:29:55.737953 10246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790321392278483-9480-0/minicluster-data/ts-1/data/instance:
uuid: "6ad613a08f2d4e4d9e10b847d7e40991"
format_stamp: "Formatted at 2026-09-25 07:29:55 on dist-test-slave-57fn"
server_key: "1abe0d1f356b5e432602e59d3dcdcf1e"
server_key_iv: "8edee9b5ea5dbb99db13370694bedc7c"
server_key_version: "encryptionkey@0"
I20260925 07:29:55.738298 10246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790321392278483-9480-0/minicluster-data/ts-1/wal/instance:
uuid: "6ad613a08f2d4e4d9e10b847d7e40991"
format_stamp: "Formatted at 2026-09-25 07:29:55 on dist-test-slave-57fn"
server_key: "1abe0d1f356b5e432602e59d3dcdcf1e"
server_key_iv: "8edee9b5ea5dbb99db13370694bedc7c"
server_key_version: "encryptionkey@0"
I20260925 07:29:55.739552 10246 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 07:29:55.740324 10261 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:29:55.740485 10246 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 07:29:55.740578 10246 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790321392278483-9480-0/minicluster-data/ts-1/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790321392278483-9480-0/minicluster-data/ts-1/wal
uuid: "6ad613a08f2d4e4d9e10b847d7e40991"
format_stamp: "Formatted at 2026-09-25 07:29:55 on dist-test-slave-57fn"
server_key: "1abe0d1f356b5e432602e59d3dcdcf1e"
server_key_iv: "8edee9b5ea5dbb99db13370694bedc7c"
server_key_version: "encryptionkey@0"
I20260925 07:29:55.740655 10246 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790321392278483-9480-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790321392278483-9480-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790321392278483-9480-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:29:55.754678 10246 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:29:55.756919 10246 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:29:55.757086 10246 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:29:55.757295 10246 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:29:55.757648 10246 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:29:55.757700 10246 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:29:55.757740 10246 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:29:55.757766 10246 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:29:55.763393 10246 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.2:39141
I20260925 07:29:55.763466 10374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.2:39141 every 8 connection(s)
I20260925 07:29:55.763727 10246 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790321392278483-9480-0/minicluster-data/ts-1/data/info.pb
Sep 25 07:29:55 dist-test-slave-57fn krb5kdc[10005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321395, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.2@KRBTEST.COM for kudu/127.9.66.62@KRBTEST.COM
I20260925 07:29:55.770418 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 10246
I20260925 07:29:55.770545 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790321392278483-9480-0/minicluster-data/ts-1/wal/instance
I20260925 07:29:55.770726 9480 external_mini_cluster.cc:1428] Setting key 309427351f4174690c28cfb717e7e534
I20260925 07:29:55.773211 10090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:29:55.764420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.2:55603 (local address 127.9.66.62:38333)
0925 07:29:55.764470 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 07:29:55.764473 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:29:55.764695 (+ 222us) server_negotiation.cc:407] Connection header received
0925 07:29:55.765602 (+ 907us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:29:55.765603 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:29:55.765618 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:29:55.765668 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:29:55.766196 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:29:55.766660 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:29:55.767560 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:29:55.767754 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:29:55.767804 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:29:55.769984 (+ 2180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:29:55.769986 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:29:55.769986 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:29:55.769992 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:29:55.771752 (+ 1760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:29:55.772146 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:29:55.772147 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:29:55.772148 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:29:55.772211 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:29:55.772471 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:29:55.772472 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:29:55.772473 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:29:55.772624 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:29:55.772686 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 07:29:55.773080 (+ 394us) server_negotiation.cc:299] Negotiation successful
0925 07:29:55.773149 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 07:29:55.773470 10377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:29:55.764418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:38333 (local address 127.9.66.2:55603)
0925 07:29:55.764576 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 07:29:55.764608 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0925 07:29:55.765452 (+ 844us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:29:55.765718 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:29:55.765721 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:29:55.765944 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:29:55.766126 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:29:55.766130 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:29:55.766732 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:29:55.766733 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:29:55.767480 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:29:55.767482 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:29:55.767822 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:29:55.767822 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:29:55.767921 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:29:55.768420 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:29:55.768428 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:29:55.769805 (+ 1377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:29:55.771854 (+ 2049us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:29:55.771859 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:29:55.771861 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:29:55.772073 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:29:55.772286 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:29:55.772287 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:29:55.772287 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:29:55.772408 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:29:55.772691 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:29:55.772697 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:29:55.772928 (+ 231us) client_negotiation.cc:769] Sending connection context
0925 07:29:55.773058 (+ 130us) client_negotiation.cc:240] Negotiation successful
0925 07:29:55.773229 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":85,"threads_started":1}
I20260925 07:29:55.773988 10375 heartbeater.cc:347] Connected to a master server at 127.9.66.62:38333
I20260925 07:29:55.774085 10375 heartbeater.cc:464] Registering TS with master...
I20260925 07:29:55.774261 10375 heartbeater.cc:511] Master 127.9.66.62:38333 requested a full tablet report, sending...
I20260925 07:29:55.774653 10050 ts_manager.cc:194] Registered new tserver with Master: 6ad613a08f2d4e4d9e10b847d7e40991 (127.9.66.2:39141)
I20260925 07:29:55.775041 10050 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.66.2@KRBTEST.COM'} at 127.9.66.2:55603
WARNING: no policy specified for kudu/127.9.66.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.66.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.66.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.3.keytab.
Entry for principal kudu/127.9.66.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.3.keytab.
I20260925 07:29:55.795689 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790321392278483-9480-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790321392278483-9480-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790321392278483-9480-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790321392278483-9480-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.66.3:0
--local_ip_for_outbound_sockets=127.9.66.3
--webserver_interface=127.9.66.3
--webserver_port=0
--tserver_master_addrs=127.9.66.62:38333
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.3.keytab
--principal=kudu/127.9.66.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:40131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:29:55.873535 10380 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:29:55.873713 10380 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:29:55.873735 10380 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:29:55.874787 10380 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:29:55.874820 10380 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:29:55.874846 10380 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.66.3
I20260925 07:29:55.876196 10380 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:40131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790321392278483-9480-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790321392278483-9480-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.3.keytab
--principal=kudu/127.9.66.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.66.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790321392278483-9480-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.66.3
--webserver_port=0
--tserver_master_addrs=127.9.66.62:38333
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10380
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.66.3
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790321392278483-9480-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:29:55.876358 10380 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:29:55.876521 10380 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:29:55.881178 10386 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:29:55.881258 10380 server_base.cc:1034] running on GCE node
W20260925 07:29:55.881146 10385 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:29:55.881142 10388 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:29:55.881609 10380 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:29:55.881881 10380 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:29:55.883045 10380 hybrid_clock.cc:648] HybridClock initialized: now 1790321395883033 us; error 28 us; skew 500 ppm
Sep 25 07:29:55 dist-test-slave-57fn krb5kdc[10005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321395, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:29:55.885331 10380 init.cc:377] Logged in from keytab as kudu/127.9.66.3@KRBTEST.COM (short username kudu)
I20260925 07:29:55.885794 10380 webserver.cc:460] Webserver started at http://127.9.66.3:37999/ using document root <none> and password file <none>
I20260925 07:29:55.886004 10380 fs_manager.cc:359] Metadata directory not provided
I20260925 07:29:55.886067 10380 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:29:55.886193 10380 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:29:55.887148 10380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790321392278483-9480-0/minicluster-data/ts-2/data/instance:
uuid: "8ed9bc9678644137af8025503d2cc4f0"
format_stamp: "Formatted at 2026-09-25 07:29:55 on dist-test-slave-57fn"
server_key: "bc0284526a5e0359d661743f7db04300"
server_key_iv: "29e04e6346189c4d683c227ff3ec39f9"
server_key_version: "encryptionkey@0"
I20260925 07:29:55.887497 10380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790321392278483-9480-0/minicluster-data/ts-2/wal/instance:
uuid: "8ed9bc9678644137af8025503d2cc4f0"
format_stamp: "Formatted at 2026-09-25 07:29:55 on dist-test-slave-57fn"
server_key: "bc0284526a5e0359d661743f7db04300"
server_key_iv: "29e04e6346189c4d683c227ff3ec39f9"
server_key_version: "encryptionkey@0"
I20260925 07:29:55.888772 10380 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 07:29:55.889485 10395 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:29:55.889657 10380 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 07:29:55.889750 10380 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790321392278483-9480-0/minicluster-data/ts-2/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790321392278483-9480-0/minicluster-data/ts-2/wal
uuid: "8ed9bc9678644137af8025503d2cc4f0"
format_stamp: "Formatted at 2026-09-25 07:29:55 on dist-test-slave-57fn"
server_key: "bc0284526a5e0359d661743f7db04300"
server_key_iv: "29e04e6346189c4d683c227ff3ec39f9"
server_key_version: "encryptionkey@0"
I20260925 07:29:55.889832 10380 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790321392278483-9480-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790321392278483-9480-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790321392278483-9480-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:29:55.901652 10380 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:29:55.903364 10380 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:29:55.903542 10380 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:29:55.903765 10380 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:29:55.904074 10380 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:29:55.904126 10380 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:29:55.904165 10380 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:29:55.904191 10380 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:29:55.910006 10380 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.3:45601
I20260925 07:29:55.910036 10508 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.3:45601 every 8 connection(s)
I20260925 07:29:55.910362 10380 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790321392278483-9480-0/minicluster-data/ts-2/data/info.pb
Sep 25 07:29:55 dist-test-slave-57fn krb5kdc[10005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321395, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.3@KRBTEST.COM for kudu/127.9.66.62@KRBTEST.COM
I20260925 07:29:55.918262 10090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:29:55.911034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.3:40689 (local address 127.9.66.62:38333)
0925 07:29:55.911106 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 07:29:55.911108 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:29:55.911270 (+ 162us) server_negotiation.cc:407] Connection header received
0925 07:29:55.912081 (+ 811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:29:55.912082 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:29:55.912091 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:29:55.912137 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:29:55.912645 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:29:55.913103 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:29:55.914075 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:29:55.914243 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:29:55.914295 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:29:55.916048 (+ 1753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:29:55.916049 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:29:55.916049 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:29:55.916055 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:29:55.917294 (+ 1239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:29:55.917637 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:29:55.917638 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:29:55.917638 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:29:55.917687 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:29:55.917868 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:29:55.917868 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:29:55.917868 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:29:55.917949 (+ 81us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:29:55.917991 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0925 07:29:55.918160 (+ 169us) server_negotiation.cc:299] Negotiation successful
0925 07:29:55.918207 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 07:29:55.918457 10511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:29:55.910995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:38333 (local address 127.9.66.3:40689)
0925 07:29:55.911145 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 07:29:55.911176 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0925 07:29:55.912006 (+ 830us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:29:55.912182 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:29:55.912185 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:29:55.912419 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:29:55.912581 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:29:55.912585 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:29:55.913171 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:29:55.913172 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:29:55.914010 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:29:55.914012 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:29:55.914308 (+ 296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:29:55.914309 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:29:55.914392 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:29:55.914764 (+ 372us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:29:55.914770 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:29:55.915974 (+ 1204us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:29:55.917356 (+ 1382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:29:55.917360 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:29:55.917361 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:29:55.917590 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:29:55.917738 (+ 148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:29:55.917738 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:29:55.917739 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:29:55.917825 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:29:55.917993 (+ 168us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:29:55.917996 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:29:55.918083 (+ 87us) client_negotiation.cc:769] Sending connection context
0925 07:29:55.918150 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 07:29:55.918243 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":89,"threads_started":1}
I20260925 07:29:55.918982 10509 heartbeater.cc:347] Connected to a master server at 127.9.66.62:38333
I20260925 07:29:55.919055 10509 heartbeater.cc:464] Registering TS with master...
I20260925 07:29:55.919169 10509 heartbeater.cc:511] Master 127.9.66.62:38333 requested a full tablet report, sending...
I20260925 07:29:55.919487 10050 ts_manager.cc:194] Registered new tserver with Master: 8ed9bc9678644137af8025503d2cc4f0 (127.9.66.3:45601)
I20260925 07:29:55.919849 10050 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.66.3@KRBTEST.COM'} at 127.9.66.3:40689
I20260925 07:29:55.919840 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 10380
I20260925 07:29:55.919966 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790321392278483-9480-0/minicluster-data/ts-2/wal/instance
I20260925 07:29:55.920111 9480 external_mini_cluster.cc:1428] Setting key 9628ae7840742973fc4b5e15579a692a
I20260925 07:29:55.920920 9480 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 07:29:55.927716 10090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:29:55.922469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57814 (local address 127.9.66.62:38333)
0925 07:29:55.922505 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 07:29:55.922506 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 07:29:55.922604 (+ 98us) server_negotiation.cc:407] Connection header received
0925 07:29:55.922653 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:29:55.922654 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:29:55.922662 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:29:55.922722 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:29:55.923247 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:29:55.923683 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:29:55.924355 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:29:55.924528 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:29:55.924582 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:29:55.925571 (+ 989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:29:55.925572 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:29:55.925572 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:29:55.925578 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:29:55.926872 (+ 1294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:29:55.927135 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:29:55.927136 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:29:55.927136 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:29:55.927180 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:29:55.927312 (+ 132us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:29:55.927312 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:29:55.927312 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:29:55.927447 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:29:55.927512 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 07:29:55.927609 (+ 97us) server_negotiation.cc:299] Negotiation successful
0925 07:29:55.927661 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 07:29:55.929651 10050 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57814:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 07:29:55.931246 10050 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 07:29:55.937709 10520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:29:55.933707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46582 (local address 127.9.66.3:45601)
0925 07:29:55.933859 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0925 07:29:55.933864 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 07:29:55.933930 (+ 66us) server_negotiation.cc:407] Connection header received
0925 07:29:55.933984 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:29:55.933986 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:29:55.934001 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:29:55.934085 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:29:55.934257 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:29:55.935283 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:29:55.936648 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:29:55.937337 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:29:55.937403 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:29:55.937454 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 07:29:55.937598 (+ 144us) server_negotiation.cc:299] Negotiation successful
0925 07:29:55.937633 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":86,"threads_started":1}
I20260925 07:29:55.937698 10518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:29:55.933466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.3:45601 (local address 127.0.0.1:46582)
0925 07:29:55.933871 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0925 07:29:55.933881 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 07:29:55.933932 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:29:55.934107 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:29:55.934110 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:29:55.934112 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:29:55.934172 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:29:55.934174 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:29:55.935453 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:29:55.935454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:29:55.936566 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:29:55.936567 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:29:55.937404 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:29:55.937405 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:29:55.937485 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:29:55.937498 (+ 13us) client_negotiation.cc:769] Sending connection context
0925 07:29:55.937584 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 07:29:55.937633 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":394,"mutex_wait_us":2,"thread_start_us":92,"threads_started":1}
I20260925 07:29:55.937942 10519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:29:55.933463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.1:42783 (local address 127.0.0.1:33744)
0925 07:29:55.933703 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0925 07:29:55.933713 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 07:29:55.933768 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:29:55.934221 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:29:55.934222 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:29:55.934224 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:29:55.934277 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:29:55.934279 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:29:55.935694 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:29:55.935695 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:29:55.936776 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:29:55.936778 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:29:55.937732 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:29:55.937732 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:29:55.937787 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:29:55.937791 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 07:29:55.937842 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 07:29:55.937883 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"mutex_wait_us":2,"thread_start_us":67,"threads_started":1}
I20260925 07:29:55.938105 10517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:29:55.933463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.2:39141 (local address 127.0.0.1:50570)
0925 07:29:55.933572 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0925 07:29:55.933607 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 07:29:55.933666 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:29:55.934374 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:29:55.934382 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:29:55.934384 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:29:55.934418 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:29:55.934419 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:29:55.935215 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:29:55.935216 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:29:55.936538 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:29:55.936540 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:29:55.937544 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:29:55.937545 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:29:55.937607 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:29:55.937611 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 07:29:55.937643 (+ 32us) client_negotiation.cc:240] Negotiation successful
0925 07:29:55.937682 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":72,"threads_started":1}
I20260925 07:29:55.938242 10521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:29:55.933716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50570 (local address 127.9.66.2:39141)
0925 07:29:55.934114 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0925 07:29:55.934119 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 07:29:55.934127 (+ 8us) server_negotiation.cc:407] Connection header received
0925 07:29:55.934163 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:29:55.934164 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:29:55.934177 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:29:55.934378 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:29:55.934478 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:29:55.935117 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:29:55.936609 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:29:55.937420 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:29:55.937739 (+ 319us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:29:55.937946 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0925 07:29:55.937972 (+ 26us) server_negotiation.cc:299] Negotiation successful
0925 07:29:55.938104 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":77,"threads_started":1}
I20260925 07:29:55.938242 10522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:29:55.933913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33744 (local address 127.9.66.1:42783)
0925 07:29:55.934078 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 07:29:55.934082 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 07:29:55.934091 (+ 9us) server_negotiation.cc:407] Connection header received
0925 07:29:55.934125 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:29:55.934127 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:29:55.934142 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:29:55.934224 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:29:55.934537 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:29:55.935599 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:29:55.936865 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:29:55.937594 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:29:55.937652 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:29:55.937702 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 07:29:55.938122 (+ 420us) server_negotiation.cc:299] Negotiation successful
0925 07:29:55.938161 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":80,"threads_started":1}
I20260925 07:29:55.938395 10443 tablet_service.cc:1467] Processing CreateTablet for tablet 41cde63574c5455a8190322361e06db1 (DEFAULT_TABLE table=test-table [id=21f95083f597416286db74171bcb69eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:29:55.938726 10443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41cde63574c5455a8190322361e06db1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:29:55.938879 10175 tablet_service.cc:1467] Processing CreateTablet for tablet 41cde63574c5455a8190322361e06db1 (DEFAULT_TABLE table=test-table [id=21f95083f597416286db74171bcb69eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:29:55.938879 10309 tablet_service.cc:1467] Processing CreateTablet for tablet 41cde63574c5455a8190322361e06db1 (DEFAULT_TABLE table=test-table [id=21f95083f597416286db74171bcb69eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:29:55.939155 10175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41cde63574c5455a8190322361e06db1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:29:55.939155 10309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41cde63574c5455a8190322361e06db1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:29:55.940975 10523 tablet_bootstrap.cc:492] T 41cde63574c5455a8190322361e06db1 P 8ed9bc9678644137af8025503d2cc4f0: Bootstrap starting.
I20260925 07:29:55.941584 10523 tablet_bootstrap.cc:654] T 41cde63574c5455a8190322361e06db1 P 8ed9bc9678644137af8025503d2cc4f0: Neither blocks nor log segments found. Creating new log.
I20260925 07:29:55.942024 10523 log.cc:824] T 41cde63574c5455a8190322361e06db1 P 8ed9bc9678644137af8025503d2cc4f0: Log is configured to *not* fsync() on all Append() calls
I20260925 07:29:55.942225 10524 tablet_bootstrap.cc:492] T 41cde63574c5455a8190322361e06db1 P 4ae4d23c4515478ca0b3f2b60c9ca78e: Bootstrap starting.
I20260925 07:29:55.942698 10525 tablet_bootstrap.cc:492] T 41cde63574c5455a8190322361e06db1 P 6ad613a08f2d4e4d9e10b847d7e40991: Bootstrap starting.
I20260925 07:29:55.942813 10524 tablet_bootstrap.cc:654] T 41cde63574c5455a8190322361e06db1 P 4ae4d23c4515478ca0b3f2b60c9ca78e: Neither blocks nor log segments found. Creating new log.
I20260925 07:29:55.942940 10523 tablet_bootstrap.cc:492] T 41cde63574c5455a8190322361e06db1 P 8ed9bc9678644137af8025503d2cc4f0: No bootstrap required, opened a new log
I20260925 07:29:55.943027 10523 ts_tablet_manager.cc:1397] T 41cde63574c5455a8190322361e06db1 P 8ed9bc9678644137af8025503d2cc4f0: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 07:29:55.943387 10525 tablet_bootstrap.cc:654] T 41cde63574c5455a8190322361e06db1 P 6ad613a08f2d4e4d9e10b847d7e40991: Neither blocks nor log segments found. Creating new log.
I20260925 07:29:55.943406 10524 log.cc:824] T 41cde63574c5455a8190322361e06db1 P 4ae4d23c4515478ca0b3f2b60c9ca78e: Log is configured to *not* fsync() on all Append() calls
I20260925 07:29:55.943979 10525 log.cc:824] T 41cde63574c5455a8190322361e06db1 P 6ad613a08f2d4e4d9e10b847d7e40991: Log is configured to *not* fsync() on all Append() calls
I20260925 07:29:55.944267 10524 tablet_bootstrap.cc:492] T 41cde63574c5455a8190322361e06db1 P 4ae4d23c4515478ca0b3f2b60c9ca78e: No bootstrap required, opened a new log
I20260925 07:29:55.944334 10524 ts_tablet_manager.cc:1397] T 41cde63574c5455a8190322361e06db1 P 4ae4d23c4515478ca0b3f2b60c9ca78e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 07:29:55.944777 10525 tablet_bootstrap.cc:492] T 41cde63574c5455a8190322361e06db1 P 6ad613a08f2d4e4d9e10b847d7e40991: No bootstrap required, opened a new log
I20260925 07:29:55.944842 10525 ts_tablet_manager.cc:1397] T 41cde63574c5455a8190322361e06db1 P 6ad613a08f2d4e4d9e10b847d7e40991: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 07:29:55.945003 10523 raft_consensus.cc:348] T 41cde63574c5455a8190322361e06db1 P 8ed9bc9678644137af8025503d2cc4f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ae4d23c4515478ca0b3f2b60c9ca78e" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 42783 } } peers { permanent_uuid: "8ed9bc9678644137af8025503d2cc4f0" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 45601 } } peers { permanent_uuid: "6ad613a08f2d4e4d9e10b847d7e40991" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 39141 } }
I20260925 07:29:55.945170 10523 raft_consensus.cc:374] T 41cde63574c5455a8190322361e06db1 P 8ed9bc9678644137af8025503d2cc4f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:29:55.945212 10523 raft_consensus.cc:729] T 41cde63574c5455a8190322361e06db1 P 8ed9bc9678644137af8025503d2cc4f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ed9bc9678644137af8025503d2cc4f0, State: Initialized, Role: FOLLOWER
I20260925 07:29:55.945317 10523 consensus_queue.cc:260] T 41cde63574c5455a8190322361e06db1 P 8ed9bc9678644137af8025503d2cc4f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ae4d23c4515478ca0b3f2b60c9ca78e" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 42783 } } peers { permanent_uuid: "8ed9bc9678644137af8025503d2cc4f0" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 45601 } } peers { permanent_uuid: "6ad613a08f2d4e4d9e10b847d7e40991" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 39141 } }
I20260925 07:29:55.945617 10523 ts_tablet_manager.cc:1428] T 41cde63574c5455a8190322361e06db1 P 8ed9bc9678644137af8025503d2cc4f0: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 07:29:55.945624 10509 heartbeater.cc:503] Master 127.9.66.62:38333 was elected leader, sending a full tablet report...
I20260925 07:29:55.946362 10524 raft_consensus.cc:348] T 41cde63574c5455a8190322361e06db1 P 4ae4d23c4515478ca0b3f2b60c9ca78e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ae4d23c4515478ca0b3f2b60c9ca78e" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 42783 } } peers { permanent_uuid: "8ed9bc9678644137af8025503d2cc4f0" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 45601 } } peers { permanent_uuid: "6ad613a08f2d4e4d9e10b847d7e40991" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 39141 } }
I20260925 07:29:55.946521 10524 raft_consensus.cc:374] T 41cde63574c5455a8190322361e06db1 P 4ae4d23c4515478ca0b3f2b60c9ca78e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:29:55.946564 10524 raft_consensus.cc:729] T 41cde63574c5455a8190322361e06db1 P 4ae4d23c4515478ca0b3f2b60c9ca78e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ae4d23c4515478ca0b3f2b60c9ca78e, State: Initialized, Role: FOLLOWER
I20260925 07:29:55.946669 10524 consensus_queue.cc:260] T 41cde63574c5455a8190322361e06db1 P 4ae4d23c4515478ca0b3f2b60c9ca78e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ae4d23c4515478ca0b3f2b60c9ca78e" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 42783 } } peers { permanent_uuid: "8ed9bc9678644137af8025503d2cc4f0" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 45601 } } peers { permanent_uuid: "6ad613a08f2d4e4d9e10b847d7e40991" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 39141 } }
I20260925 07:29:55.946923 10524 ts_tablet_manager.cc:1428] T 41cde63574c5455a8190322361e06db1 P 4ae4d23c4515478ca0b3f2b60c9ca78e: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 07:29:55.946832 10525 raft_consensus.cc:348] T 41cde63574c5455a8190322361e06db1 P 6ad613a08f2d4e4d9e10b847d7e40991 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ae4d23c4515478ca0b3f2b60c9ca78e" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 42783 } } peers { permanent_uuid: "8ed9bc9678644137af8025503d2cc4f0" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 45601 } } peers { permanent_uuid: "6ad613a08f2d4e4d9e10b847d7e40991" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 39141 } }
I20260925 07:29:55.946935 10241 heartbeater.cc:503] Master 127.9.66.62:38333 was elected leader, sending a full tablet report...
I20260925 07:29:55.946978 10525 raft_consensus.cc:374] T 41cde63574c5455a8190322361e06db1 P 6ad613a08f2d4e4d9e10b847d7e40991 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:29:55.947012 10525 raft_consensus.cc:729] T 41cde63574c5455a8190322361e06db1 P 6ad613a08f2d4e4d9e10b847d7e40991 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ad613a08f2d4e4d9e10b847d7e40991, State: Initialized, Role: FOLLOWER
I20260925 07:29:55.947127 10525 consensus_queue.cc:260] T 41cde63574c5455a8190322361e06db1 P 6ad613a08f2d4e4d9e10b847d7e40991 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ae4d23c4515478ca0b3f2b60c9ca78e" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 42783 } } peers { permanent_uuid: "8ed9bc9678644137af8025503d2cc4f0" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 45601 } } peers { permanent_uuid: "6ad613a08f2d4e4d9e10b847d7e40991" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 39141 } }
I20260925 07:29:55.947376 10375 heartbeater.cc:503] Master 127.9.66.62:38333 was elected leader, sending a full tablet report...
I20260925 07:29:55.947394 10525 ts_tablet_manager.cc:1428] T 41cde63574c5455a8190322361e06db1 P 6ad613a08f2d4e4d9e10b847d7e40991: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 07:29:55.969625 10529 raft_consensus.cc:482] T 41cde63574c5455a8190322361e06db1 P 8ed9bc9678644137af8025503d2cc4f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 07:29:55.969754 10529 raft_consensus.cc:504] T 41cde63574c5455a8190322361e06db1 P 8ed9bc9678644137af8025503d2cc4f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ae4d23c4515478ca0b3f2b60c9ca78e" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 42783 } } peers { permanent_uuid: "8ed9bc9678644137af8025503d2cc4f0" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 45601 } } peers { permanent_uuid: "6ad613a08f2d4e4d9e10b847d7e40991" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 39141 } }
I20260925 07:29:55.970090 10529 leader_election.cc:288] T 41cde63574c5455a8190322361e06db1 P 8ed9bc9678644137af8025503d2cc4f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4ae4d23c4515478ca0b3f2b60c9ca78e (127.9.66.1:42783), 6ad613a08f2d4e4d9e10b847d7e40991 (127.9.66.2:39141)
I20260925 07:29:55.972824 10511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:29:55.970273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.2:39141 (local address 127.9.66.3:39265)
0925 07:29:55.970331 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0925 07:29:55.970343 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 07:29:55.970387 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:29:55.970517 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:29:55.970517 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:29:55.970519 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:29:55.970598 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:29:55.970600 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:29:55.971270 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:29:55.971272 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:29:55.972037 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:29:55.972038 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:29:55.972586 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:29:55.972587 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:29:55.972666 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:29:55.972669 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 07:29:55.972717 (+ 48us) client_negotiation.cc:240] Negotiation successful
0925 07:29:55.972760 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":27}
I20260925 07:29:55.972877 10521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:29:55.970356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.3:39265 (local address 127.9.66.2:39141)
0925 07:29:55.970396 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 07:29:55.970398 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:29:55.970407 (+ 9us) server_negotiation.cc:407] Connection header received
0925 07:29:55.970432 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:29:55.970433 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:29:55.970442 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:29:55.970506 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:29:55.970669 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:29:55.971193 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:29:55.972118 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:29:55.972494 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:29:55.972558 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:29:55.972564 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0925 07:29:55.972763 (+ 199us) server_negotiation.cc:299] Negotiation successful
0925 07:29:55.972801 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 07:29:55.972936 10522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:29:55.970347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.3:45573 (local address 127.9.66.1:42783)
0925 07:29:55.970396 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 07:29:55.970398 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:29:55.970572 (+ 174us) server_negotiation.cc:407] Connection header received
0925 07:29:55.970611 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:29:55.970612 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:29:55.970624 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:29:55.970674 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:29:55.970792 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:29:55.971266 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:29:55.972166 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:29:55.972505 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:29:55.972617 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:29:55.972626 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 07:29:55.972806 (+ 180us) server_negotiation.cc:299] Negotiation successful
0925 07:29:55.972849 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 07:29:55.973074 10329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41cde63574c5455a8190322361e06db1" candidate_uuid: "8ed9bc9678644137af8025503d2cc4f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ad613a08f2d4e4d9e10b847d7e40991" is_pre_election: true
I20260925 07:29:55.973142 10532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:29:55.970293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.1:42783 (local address 127.9.66.3:45573)
0925 07:29:55.970497 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0925 07:29:55.970507 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 07:29:55.970566 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:29:55.970674 (+ 108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:29:55.970675 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:29:55.970676 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:29:55.970733 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:29:55.970734 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:29:55.971344 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:29:55.971345 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:29:55.972060 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:29:55.972062 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:29:55.972634 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:29:55.972635 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:29:55.972714 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:29:55.972717 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 07:29:55.973015 (+ 298us) client_negotiation.cc:240] Negotiation successful
0925 07:29:55.973068 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"mutex_wait_us":55,"thread_start_us":94,"threads_started":1}
I20260925 07:29:55.973186 10329 raft_consensus.cc:2454] T 41cde63574c5455a8190322361e06db1 P 6ad613a08f2d4e4d9e10b847d7e40991 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ed9bc9678644137af8025503d2cc4f0 in term 0.
I20260925 07:29:55.973394 10195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41cde63574c5455a8190322361e06db1" candidate_uuid: "8ed9bc9678644137af8025503d2cc4f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ae4d23c4515478ca0b3f2b60c9ca78e" is_pre_election: true
I20260925 07:29:55.973487 10195 raft_consensus.cc:2454] T 41cde63574c5455a8190322361e06db1 P 4ae4d23c4515478ca0b3f2b60c9ca78e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ed9bc9678644137af8025503d2cc4f0 in term 0.
I20260925 07:29:55.973634 10397 leader_election.cc:302] T 41cde63574c5455a8190322361e06db1 P 8ed9bc9678644137af8025503d2cc4f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ad613a08f2d4e4d9e10b847d7e40991, 8ed9bc9678644137af8025503d2cc4f0; no voters:
I20260925 07:29:55.973754 10529 raft_consensus.cc:2781] T 41cde63574c5455a8190322361e06db1 P 8ed9bc9678644137af8025503d2cc4f0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 07:29:55.973839 10529 raft_consensus.cc:482] T 41cde63574c5455a8190322361e06db1 P 8ed9bc9678644137af8025503d2cc4f0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 07:29:55.973875 10529 raft_consensus.cc:3037] T 41cde63574c5455a8190322361e06db1 P 8ed9bc9678644137af8025503d2cc4f0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:29:55.974481 10529 raft_consensus.cc:504] T 41cde63574c5455a8190322361e06db1 P 8ed9bc9678644137af8025503d2cc4f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ae4d23c4515478ca0b3f2b60c9ca78e" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 42783 } } peers { permanent_uuid: "8ed9bc9678644137af8025503d2cc4f0" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 45601 } } peers { permanent_uuid: "6ad613a08f2d4e4d9e10b847d7e40991" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 39141 } }
I20260925 07:29:55.974619 10529 leader_election.cc:288] T 41cde63574c5455a8190322361e06db1 P 8ed9bc9678644137af8025503d2cc4f0 [CANDIDATE]: Term 1 election: Requested vote from peers 4ae4d23c4515478ca0b3f2b60c9ca78e (127.9.66.1:42783), 6ad613a08f2d4e4d9e10b847d7e40991 (127.9.66.2:39141)
I20260925 07:29:55.974748 10329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41cde63574c5455a8190322361e06db1" candidate_uuid: "8ed9bc9678644137af8025503d2cc4f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ad613a08f2d4e4d9e10b847d7e40991"
I20260925 07:29:55.974747 10195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41cde63574c5455a8190322361e06db1" candidate_uuid: "8ed9bc9678644137af8025503d2cc4f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ae4d23c4515478ca0b3f2b60c9ca78e"
I20260925 07:29:55.974814 10329 raft_consensus.cc:3037] T 41cde63574c5455a8190322361e06db1 P 6ad613a08f2d4e4d9e10b847d7e40991 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:29:55.974821 10195 raft_consensus.cc:3037] T 41cde63574c5455a8190322361e06db1 P 4ae4d23c4515478ca0b3f2b60c9ca78e [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:29:55.975637 10329 raft_consensus.cc:2454] T 41cde63574c5455a8190322361e06db1 P 6ad613a08f2d4e4d9e10b847d7e40991 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ed9bc9678644137af8025503d2cc4f0 in term 1.
I20260925 07:29:55.975637 10195 raft_consensus.cc:2454] T 41cde63574c5455a8190322361e06db1 P 4ae4d23c4515478ca0b3f2b60c9ca78e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ed9bc9678644137af8025503d2cc4f0 in term 1.
I20260925 07:29:55.975801 10397 leader_election.cc:302] T 41cde63574c5455a8190322361e06db1 P 8ed9bc9678644137af8025503d2cc4f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ad613a08f2d4e4d9e10b847d7e40991, 8ed9bc9678644137af8025503d2cc4f0; no voters:
I20260925 07:29:55.975900 10529 raft_consensus.cc:2781] T 41cde63574c5455a8190322361e06db1 P 8ed9bc9678644137af8025503d2cc4f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:29:55.976017 10529 raft_consensus.cc:686] T 41cde63574c5455a8190322361e06db1 P 8ed9bc9678644137af8025503d2cc4f0 [term 1 LEADER]: Becoming Leader. State: Replica: 8ed9bc9678644137af8025503d2cc4f0, State: Running, Role: LEADER
I20260925 07:29:55.976100 10529 consensus_queue.cc:237] T 41cde63574c5455a8190322361e06db1 P 8ed9bc9678644137af8025503d2cc4f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ae4d23c4515478ca0b3f2b60c9ca78e" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 42783 } } peers { permanent_uuid: "8ed9bc9678644137af8025503d2cc4f0" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 45601 } } peers { permanent_uuid: "6ad613a08f2d4e4d9e10b847d7e40991" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 39141 } }
I20260925 07:29:55.976780 10049 catalog_manager.cc:5495] T 41cde63574c5455a8190322361e06db1 P 8ed9bc9678644137af8025503d2cc4f0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ed9bc9678644137af8025503d2cc4f0 (127.9.66.3). New cstate: current_term: 1 leader_uuid: "8ed9bc9678644137af8025503d2cc4f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ae4d23c4515478ca0b3f2b60c9ca78e" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 42783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ed9bc9678644137af8025503d2cc4f0" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 45601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6ad613a08f2d4e4d9e10b847d7e40991" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 39141 } health_report { overall_health: UNKNOWN } } }
Sep 25 07:29:55 dist-test-slave-57fn krb5kdc[10005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321395, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.66.1@KRBTEST.COM
I20260925 07:29:55.989952 10522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:29:55.983214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33756 (local address 127.9.66.1:42783)
0925 07:29:55.983266 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 07:29:55.983268 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:29:55.983406 (+ 138us) server_negotiation.cc:407] Connection header received
0925 07:29:55.983461 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:29:55.983461 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:29:55.983468 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:29:55.983513 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:29:55.984067 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:29:55.984478 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:29:55.985230 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:29:55.985391 (+ 161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:29:55.985434 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:29:55.987709 (+ 2275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:29:55.987714 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:29:55.987714 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:29:55.987720 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:29:55.989034 (+ 1314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:29:55.989321 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:29:55.989324 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:29:55.989324 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:29:55.989371 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:29:55.989569 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:29:55.989569 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:29:55.989570 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:29:55.989683 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:29:55.989780 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 07:29:55.989835 (+ 55us) server_negotiation.cc:299] Negotiation successful
0925 07:29:55.989901 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
Sep 25 07:29:55 dist-test-slave-57fn krb5kdc[10005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321395, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 07:29:56 dist-test-slave-57fn krb5kdc[10005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321395, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.9.66.1@KRBTEST.COM
I20260925 07:29:56.011513 10522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:29:56.004750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33772 (local address 127.9.66.1:42783)
0925 07:29:56.004810 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 07:29:56.004813 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:29:56.004822 (+ 9us) server_negotiation.cc:407] Connection header received
0925 07:29:56.004897 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:29:56.004898 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:29:56.004908 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:29:56.004980 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:29:56.005530 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:29:56.006015 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:29:56.006755 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:29:56.006937 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:29:56.007008 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:29:56.009012 (+ 2004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:29:56.009013 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:29:56.009013 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:29:56.009018 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:29:56.010444 (+ 1426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:29:56.010739 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:29:56.010740 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:29:56.010740 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:29:56.010805 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:29:56.010996 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:29:56.010997 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:29:56.010997 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:29:56.011105 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:29:56.011199 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 07:29:56.011360 (+ 161us) server_negotiation.cc:299] Negotiation successful
0925 07:29:56.011432 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20260925 07:29:56.014245 10376 tablet.cc:2369] T 41cde63574c5455a8190322361e06db1 P 6ad613a08f2d4e4d9e10b847d7e40991: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 25 07:29:56 dist-test-slave-57fn krb5kdc[10005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321396, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 07:29:56 dist-test-slave-57fn krb5kdc[10005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321396, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.66.1@KRBTEST.COM
I20260925 07:29:56.032305 10522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:29:56.026022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33784 (local address 127.9.66.1:42783)
0925 07:29:56.026082 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 07:29:56.026085 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:29:56.026188 (+ 103us) server_negotiation.cc:407] Connection header received
0925 07:29:56.026266 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:29:56.026271 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:29:56.026279 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:29:56.026338 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:29:56.026840 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:29:56.027295 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:29:56.028027 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:29:56.028208 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:29:56.028289 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:29:56.030350 (+ 2061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:29:56.030351 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:29:56.030352 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:29:56.030358 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:29:56.031483 (+ 1125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:29:56.031760 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:29:56.031761 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:29:56.031761 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:29:56.031806 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:29:56.031948 (+ 142us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:29:56.031948 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:29:56.031949 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:29:56.032020 (+ 71us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:29:56.032083 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 07:29:56.032172 (+ 89us) server_negotiation.cc:299] Negotiation successful
0925 07:29:56.032221 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
W20260925 07:29:56.032784 10155 tablet_service.cc:2908] Rejecting scan request for tablet 41cde63574c5455a8190322361e06db1: Uninitialized: clean time has not yet been initialized
I20260925 07:29:56.033577 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 10112
I20260925 07:29:56.038765 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 10246
I20260925 07:29:56.043059 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 10380
I20260925 07:29:56.047461 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 10018
2026-09-25T07:29:56Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (1882 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790321392278483-9480-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 07:29:56 dist-test-slave-57fn krb5kdc[10547](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 07:29:56 dist-test-slave-57fn krb5kdc[10547](info): set up 2 sockets
Sep 25 07:29:56 dist-test-slave-57fn krb5kdc[10547](info): commencing operation
krb5kdc: starting...
W20260925 07:29:57.101022 9480 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790321392278483-9480-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790321392278483-9480-0/krb5kdc/test-user.keytab.
Sep 25 07:29:57 dist-test-slave-57fn krb5kdc[10547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321397, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T07:29:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T07:29:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.66.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.66.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.66.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab.
Entry for principal kudu/127.9.66.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab.
I20260925 07:29:57.214401 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790321392278483-9480-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790321392278483-9480-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790321392278483-9480-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.66.62:37099
--webserver_interface=127.9.66.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab
--principal=kudu/127.9.66.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:33391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.66.62:37099
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:29:57.294117 10560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:29:57.294350 10560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:29:57.294386 10560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:29:57.295478 10560 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:29:57.295567 10560 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 07:29:57.295589 10560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:29:57.295611 10560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:29:57.295651 10560 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:29:57.295673 10560 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:29:57.297118 10560 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:33391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790321392278483-9480-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790321392278483-9480-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.66.62:37099
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab
--principal=kudu/127.9.66.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.66.62:37099
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.66.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10560
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790321392278483-9480-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:29:57.297389 10560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:29:57.297631 10560 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:29:57.302183 10568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:29:57.302165 10565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:29:57.302177 10566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:29:57.302491 10560 server_base.cc:1034] running on GCE node
I20260925 07:29:57.302742 10560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:29:57.303012 10560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:29:57.304219 10560 hybrid_clock.cc:648] HybridClock initialized: now 1790321397304217 us; error 45 us; skew 500 ppm
Sep 25 07:29:57 dist-test-slave-57fn krb5kdc[10547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321397, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:29:57.306668 10560 init.cc:377] Logged in from keytab as kudu/127.9.66.62@KRBTEST.COM (short username kudu)
I20260925 07:29:57.307174 10560 webserver.cc:460] Webserver started at http://127.9.66.62:37169/ using document root <none> and password file <none>
I20260925 07:29:57.307418 10560 fs_manager.cc:359] Metadata directory not provided
I20260925 07:29:57.307484 10560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:29:57.307626 10560 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:29:57.308631 10560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790321392278483-9480-0/minicluster-data/master-0/data/instance:
uuid: "42650a67401745e9be59eb4711a81af7"
format_stamp: "Formatted at 2026-09-25 07:29:57 on dist-test-slave-57fn"
server_key: "15e2aeadf034fc7be453c674a751aaa6"
server_key_iv: "927f8858d7b0474005a75751cf2d7c0b"
server_key_version: "encryptionkey@0"
I20260925 07:29:57.308986 10560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790321392278483-9480-0/minicluster-data/master-0/wal/instance:
uuid: "42650a67401745e9be59eb4711a81af7"
format_stamp: "Formatted at 2026-09-25 07:29:57 on dist-test-slave-57fn"
server_key: "15e2aeadf034fc7be453c674a751aaa6"
server_key_iv: "927f8858d7b0474005a75751cf2d7c0b"
server_key_version: "encryptionkey@0"
I20260925 07:29:57.310400 10560 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 07:29:57.311126 10575 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:29:57.311298 10560 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 07:29:57.311421 10560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790321392278483-9480-0/minicluster-data/master-0/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790321392278483-9480-0/minicluster-data/master-0/wal
uuid: "42650a67401745e9be59eb4711a81af7"
format_stamp: "Formatted at 2026-09-25 07:29:57 on dist-test-slave-57fn"
server_key: "15e2aeadf034fc7be453c674a751aaa6"
server_key_iv: "927f8858d7b0474005a75751cf2d7c0b"
server_key_version: "encryptionkey@0"
I20260925 07:29:57.311494 10560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790321392278483-9480-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790321392278483-9480-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790321392278483-9480-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:29:57.327507 10560 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:29:57.329337 10560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:29:57.329530 10560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:29:57.333981 10560 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.62:37099
I20260925 07:29:57.334036 10637 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.62:37099 every 8 connection(s)
I20260925 07:29:57.334331 10560 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
I20260925 07:29:57.334920 10638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:29:57.337077 10638 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42650a67401745e9be59eb4711a81af7: Bootstrap starting.
I20260925 07:29:57.337724 10638 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 42650a67401745e9be59eb4711a81af7: Neither blocks nor log segments found. Creating new log.
I20260925 07:29:57.338011 10638 log.cc:824] T 00000000000000000000000000000000 P 42650a67401745e9be59eb4711a81af7: Log is configured to *not* fsync() on all Append() calls
I20260925 07:29:57.338691 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 10560
I20260925 07:29:57.338765 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790321392278483-9480-0/minicluster-data/master-0/wal/instance
I20260925 07:29:57.338762 10638 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42650a67401745e9be59eb4711a81af7: No bootstrap required, opened a new log
I20260925 07:29:57.338932 9480 external_mini_cluster.cc:1428] Setting key 3fc88487da1ed651ce79ec5e8d7b808c
I20260925 07:29:57.340960 10638 raft_consensus.cc:348] T 00000000000000000000000000000000 P 42650a67401745e9be59eb4711a81af7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42650a67401745e9be59eb4711a81af7" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 37099 } }
I20260925 07:29:57.341138 10638 raft_consensus.cc:374] T 00000000000000000000000000000000 P 42650a67401745e9be59eb4711a81af7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:29:57.341203 10638 raft_consensus.cc:729] T 00000000000000000000000000000000 P 42650a67401745e9be59eb4711a81af7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42650a67401745e9be59eb4711a81af7, State: Initialized, Role: FOLLOWER
I20260925 07:29:57.341362 10638 consensus_queue.cc:260] T 00000000000000000000000000000000 P 42650a67401745e9be59eb4711a81af7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42650a67401745e9be59eb4711a81af7" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 37099 } }
I20260925 07:29:57.341454 10638 raft_consensus.cc:388] T 00000000000000000000000000000000 P 42650a67401745e9be59eb4711a81af7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 07:29:57.341496 10638 raft_consensus.cc:482] T 00000000000000000000000000000000 P 42650a67401745e9be59eb4711a81af7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 07:29:57.341569 10638 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 42650a67401745e9be59eb4711a81af7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:29:57.342080 10638 raft_consensus.cc:504] T 00000000000000000000000000000000 P 42650a67401745e9be59eb4711a81af7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42650a67401745e9be59eb4711a81af7" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 37099 } }
I20260925 07:29:57.342197 10638 leader_election.cc:302] T 00000000000000000000000000000000 P 42650a67401745e9be59eb4711a81af7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 42650a67401745e9be59eb4711a81af7; no voters:
I20260925 07:29:57.342383 10638 leader_election.cc:288] T 00000000000000000000000000000000 P 42650a67401745e9be59eb4711a81af7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 07:29:57.342417 10643 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 42650a67401745e9be59eb4711a81af7 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:29:57.342592 10643 raft_consensus.cc:686] T 00000000000000000000000000000000 P 42650a67401745e9be59eb4711a81af7 [term 1 LEADER]: Becoming Leader. State: Replica: 42650a67401745e9be59eb4711a81af7, State: Running, Role: LEADER
I20260925 07:29:57.342618 10638 sys_catalog.cc:564] T 00000000000000000000000000000000 P 42650a67401745e9be59eb4711a81af7 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:29:57.342742 10643 consensus_queue.cc:237] T 00000000000000000000000000000000 P 42650a67401745e9be59eb4711a81af7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42650a67401745e9be59eb4711a81af7" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 37099 } }
I20260925 07:29:57.343081 10644 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42650a67401745e9be59eb4711a81af7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "42650a67401745e9be59eb4711a81af7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42650a67401745e9be59eb4711a81af7" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 37099 } } }
I20260925 07:29:57.343113 10645 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42650a67401745e9be59eb4711a81af7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 42650a67401745e9be59eb4711a81af7. Latest consensus state: current_term: 1 leader_uuid: "42650a67401745e9be59eb4711a81af7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42650a67401745e9be59eb4711a81af7" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 37099 } } }
I20260925 07:29:57.343281 10644 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42650a67401745e9be59eb4711a81af7 [sys.catalog]: This master's current role is: LEADER
I20260925 07:29:57.343295 10645 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42650a67401745e9be59eb4711a81af7 [sys.catalog]: This master's current role is: LEADER
I20260925 07:29:57.344161 10652 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 07:29:57.344839 10652 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 07:29:57.346431 10652 catalog_manager.cc:1347] Generated new cluster ID: c0ee808c62664ca5ad6a40451a4a5dbc
I20260925 07:29:57.346482 10652 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 07:29:57 dist-test-slave-57fn krb5kdc[10547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321397, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.66.62@KRBTEST.COM
I20260925 07:29:57.350638 10642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:29:57.339498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46150 (local address 127.9.66.62:37099)
0925 07:29:57.339670 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0925 07:29:57.339679 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 07:29:57.339710 (+ 31us) server_negotiation.cc:407] Connection header received
0925 07:29:57.340957 (+ 1247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:29:57.340963 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:29:57.341011 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:29:57.341151 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:29:57.341960 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:29:57.343313 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:29:57.344327 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:29:57.344789 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:29:57.344908 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:29:57.347419 (+ 2511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:29:57.347423 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:29:57.347423 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:29:57.347434 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:29:57.349083 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:29:57.349453 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:29:57.349455 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:29:57.349456 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:29:57.349625 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:29:57.349832 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:29:57.349833 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:29:57.349833 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:29:57.350070 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:29:57.350254 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0925 07:29:57.350330 (+ 76us) server_negotiation.cc:299] Negotiation successful
0925 07:29:57.350466 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":73,"threads_started":1}
I20260925 07:29:57.382196 10652 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 07:29:57.382741 10652 catalog_manager.cc:1504] Loading token signing keys...
I20260925 07:29:57.389423 10652 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 42650a67401745e9be59eb4711a81af7: Generated new TSK 0
I20260925 07:29:57.389619 10652 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.66.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.66.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.66.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.1.keytab.
Entry for principal kudu/127.9.66.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.1.keytab.
I20260925 07:29:57.424423 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790321392278483-9480-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790321392278483-9480-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790321392278483-9480-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790321392278483-9480-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.66.1:0
--local_ip_for_outbound_sockets=127.9.66.1
--webserver_interface=127.9.66.1
--webserver_port=0
--tserver_master_addrs=127.9.66.62:37099
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.1.keytab
--principal=kudu/127.9.66.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:33391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:29:57.519192 10664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:29:57.519373 10664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:29:57.519397 10664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:29:57.520444 10664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:29:57.520483 10664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:29:57.520506 10664 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 07:29:57.520525 10664 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.66.1
I20260925 07:29:57.521890 10664 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:33391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790321392278483-9480-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790321392278483-9480-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.1.keytab
--principal=kudu/127.9.66.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.66.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790321392278483-9480-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.66.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.66.62:37099
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10664
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.66.1
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790321392278483-9480-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:29:57.522075 10664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:29:57.522246 10664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:29:57.527025 10669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:29:57.527091 10672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:29:57.527238 10664 server_base.cc:1034] running on GCE node
W20260925 07:29:57.527025 10670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:29:57.527500 10664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:29:57.527784 10664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:29:57.528929 10664 hybrid_clock.cc:648] HybridClock initialized: now 1790321397528920 us; error 29 us; skew 500 ppm
Sep 25 07:29:57 dist-test-slave-57fn krb5kdc[10547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321397, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:29:57.531355 10664 init.cc:377] Logged in from keytab as kudu/127.9.66.1@KRBTEST.COM (short username kudu)
I20260925 07:29:57.531798 10664 webserver.cc:460] Webserver started at http://127.9.66.1:41171/ using document root <none> and password file <none>
I20260925 07:29:57.532016 10664 fs_manager.cc:359] Metadata directory not provided
I20260925 07:29:57.532058 10664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:29:57.532192 10664 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:29:57.533146 10664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790321392278483-9480-0/minicluster-data/ts-0/data/instance:
uuid: "724be09c3ade4d0d9043cce4e15a7e26"
format_stamp: "Formatted at 2026-09-25 07:29:57 on dist-test-slave-57fn"
server_key: "36ce65540d8bc5ac06a66958f0dd5a2d"
server_key_iv: "e93643e2e4ddc606b4c31605916fe7ed"
server_key_version: "encryptionkey@0"
I20260925 07:29:57.533538 10664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790321392278483-9480-0/minicluster-data/ts-0/wal/instance:
uuid: "724be09c3ade4d0d9043cce4e15a7e26"
format_stamp: "Formatted at 2026-09-25 07:29:57 on dist-test-slave-57fn"
server_key: "36ce65540d8bc5ac06a66958f0dd5a2d"
server_key_iv: "e93643e2e4ddc606b4c31605916fe7ed"
server_key_version: "encryptionkey@0"
I20260925 07:29:57.534807 10664 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:29:57.535528 10679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:29:57.535713 10664 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 07:29:57.535811 10664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790321392278483-9480-0/minicluster-data/ts-0/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790321392278483-9480-0/minicluster-data/ts-0/wal
uuid: "724be09c3ade4d0d9043cce4e15a7e26"
format_stamp: "Formatted at 2026-09-25 07:29:57 on dist-test-slave-57fn"
server_key: "36ce65540d8bc5ac06a66958f0dd5a2d"
server_key_iv: "e93643e2e4ddc606b4c31605916fe7ed"
server_key_version: "encryptionkey@0"
I20260925 07:29:57.535895 10664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790321392278483-9480-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790321392278483-9480-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790321392278483-9480-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:29:57.557376 10664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:29:57.559034 10664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:29:57.559245 10664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:29:57.559729 10664 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:29:57.559796 10664 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:29:57.559840 10664 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:29:57.559871 10664 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 07:29:57 dist-test-slave-57fn krb5kdc[10547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321397, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.1@KRBTEST.COM for kudu/127.9.66.62@KRBTEST.COM
I20260925 07:29:57.567116 10664 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.1:40653
I20260925 07:29:57.567160 10794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.1:40653 every 8 connection(s)
I20260925 07:29:57.567656 10664 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790321392278483-9480-0/minicluster-data/ts-0/data/info.pb
I20260925 07:29:57.569478 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 10664
I20260925 07:29:57.569597 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790321392278483-9480-0/minicluster-data/ts-0/wal/instance
I20260925 07:29:57.569788 9480 external_mini_cluster.cc:1428] Setting key 1ce44f7e27a1ef862c8c4372daf77007
I20260925 07:29:57.569988 10642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:29:57.560359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.1:46913 (local address 127.9.66.62:37099)
0925 07:29:57.560437 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0925 07:29:57.560439 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:29:57.560624 (+ 185us) server_negotiation.cc:407] Connection header received
0925 07:29:57.561100 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:29:57.561102 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:29:57.561120 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:29:57.561192 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:29:57.561927 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:29:57.562431 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:29:57.563412 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:29:57.563619 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:29:57.563725 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:29:57.566993 (+ 3268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:29:57.566996 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:29:57.566996 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:29:57.567004 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:29:57.568622 (+ 1618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:29:57.569000 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:29:57.569001 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:29:57.569001 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:29:57.569051 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:29:57.569265 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:29:57.569265 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:29:57.569266 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:29:57.569365 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:29:57.569413 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 07:29:57.569765 (+ 352us) server_negotiation.cc:299] Negotiation successful
0925 07:29:57.569890 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 07:29:57.570158 10688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:29:57.560352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:37099 (local address 127.9.66.1:46913)
0925 07:29:57.560514 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 07:29:57.560547 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 07:29:57.560993 (+ 446us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:29:57.561279 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:29:57.561283 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:29:57.561648 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:29:57.561827 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:29:57.561832 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:29:57.562527 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:29:57.562528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:29:57.563318 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:29:57.563320 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:29:57.563731 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:29:57.563733 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:29:57.563874 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:29:57.564377 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:29:57.564388 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:29:57.566862 (+ 2474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:29:57.568700 (+ 1838us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:29:57.568706 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:29:57.568707 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:29:57.568920 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:29:57.569107 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:29:57.569107 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:29:57.569108 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:29:57.569208 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:29:57.569426 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:29:57.569433 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:29:57.569632 (+ 199us) client_negotiation.cc:769] Sending connection context
0925 07:29:57.569751 (+ 119us) client_negotiation.cc:240] Negotiation successful
0925 07:29:57.569919 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":100,"threads_started":1}
I20260925 07:29:57.571887 10795 heartbeater.cc:347] Connected to a master server at 127.9.66.62:37099
I20260925 07:29:57.571991 10795 heartbeater.cc:464] Registering TS with master...
I20260925 07:29:57.572153 10795 heartbeater.cc:511] Master 127.9.66.62:37099 requested a full tablet report, sending...
I20260925 07:29:57.572618 10591 ts_manager.cc:194] Registered new tserver with Master: 724be09c3ade4d0d9043cce4e15a7e26 (127.9.66.1:40653)
I20260925 07:29:57.573539 10591 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.66.1@KRBTEST.COM'} at 127.9.66.1:46913
I20260925 07:29:57.578186 10642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:29:57.572072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.1:52123 (local address 127.9.66.62:37099)
0925 07:29:57.572139 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 07:29:57.572142 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:29:57.572159 (+ 17us) server_negotiation.cc:407] Connection header received
0925 07:29:57.572181 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:29:57.572182 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:29:57.572192 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:29:57.572249 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:29:57.572666 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:29:57.573431 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:29:57.574566 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:29:57.574735 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:29:57.574788 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:29:57.575785 (+ 997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:29:57.575786 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:29:57.575786 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:29:57.575791 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:29:57.577072 (+ 1281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:29:57.577464 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:29:57.577464 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:29:57.577465 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:29:57.577546 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:29:57.577738 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:29:57.577739 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:29:57.577739 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:29:57.577825 (+ 86us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:29:57.577872 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 07:29:57.578059 (+ 187us) server_negotiation.cc:299] Negotiation successful
0925 07:29:57.578132 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":49}
I20260925 07:29:57.578281 10802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:29:57.571838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:37099 (local address 127.9.66.1:52123)
0925 07:29:57.571996 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 07:29:57.572008 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 07:29:57.572060 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:29:57.572295 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:29:57.572296 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:29:57.572511 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:29:57.572583 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:29:57.572585 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:29:57.573562 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:29:57.573563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:29:57.574483 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:29:57.574486 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:29:57.574810 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:29:57.574811 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:29:57.574883 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:29:57.575252 (+ 369us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:29:57.575258 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:29:57.575700 (+ 442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:29:57.577154 (+ 1454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:29:57.577155 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:29:57.577156 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:29:57.577386 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:29:57.577606 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:29:57.577607 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:29:57.577608 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:29:57.577672 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:29:57.577894 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:29:57.577895 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:29:57.577950 (+ 55us) client_negotiation.cc:769] Sending connection context
0925 07:29:57.578045 (+ 95us) client_negotiation.cc:240] Negotiation successful
0925 07:29:57.578170 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":88,"threads_started":1}
WARNING: no policy specified for kudu/127.9.66.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.66.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.66.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.2.keytab.
Entry for principal kudu/127.9.66.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.2.keytab.
I20260925 07:29:57.594434 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790321392278483-9480-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790321392278483-9480-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790321392278483-9480-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790321392278483-9480-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.66.2:0
--local_ip_for_outbound_sockets=127.9.66.2
--webserver_interface=127.9.66.2
--webserver_port=0
--tserver_master_addrs=127.9.66.62:37099
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.2.keytab
--principal=kudu/127.9.66.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:33391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:29:57.673180 10804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:29:57.673377 10804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:29:57.673403 10804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:29:57.674573 10804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:29:57.674618 10804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:29:57.674640 10804 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 07:29:57.674659 10804 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.66.2
I20260925 07:29:57.676035 10804 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:33391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790321392278483-9480-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790321392278483-9480-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.2.keytab
--principal=kudu/127.9.66.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.66.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790321392278483-9480-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.66.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.66.62:37099
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10804
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.66.2
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790321392278483-9480-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:29:57.676213 10804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:29:57.676381 10804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:29:57.681200 10812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:29:57.681236 10810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:29:57.681417 10804 server_base.cc:1034] running on GCE node
W20260925 07:29:57.681208 10809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:29:57.681713 10804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:29:57.682003 10804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:29:57.683153 10804 hybrid_clock.cc:648] HybridClock initialized: now 1790321397683133 us; error 53 us; skew 500 ppm
Sep 25 07:29:57 dist-test-slave-57fn krb5kdc[10547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321397, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:29:57.685626 10804 init.cc:377] Logged in from keytab as kudu/127.9.66.2@KRBTEST.COM (short username kudu)
I20260925 07:29:57.686096 10804 webserver.cc:460] Webserver started at http://127.9.66.2:33993/ using document root <none> and password file <none>
I20260925 07:29:57.686338 10804 fs_manager.cc:359] Metadata directory not provided
I20260925 07:29:57.686410 10804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:29:57.686564 10804 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:29:57.687573 10804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790321392278483-9480-0/minicluster-data/ts-1/data/instance:
uuid: "25ae7fefa24c400f8a5e752d3845fdea"
format_stamp: "Formatted at 2026-09-25 07:29:57 on dist-test-slave-57fn"
server_key: "39a7aecebfc8f41b566a0e2996dcd22d"
server_key_iv: "794b9128f462edf476773eb4abca5ec3"
server_key_version: "encryptionkey@0"
I20260925 07:29:57.687925 10804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790321392278483-9480-0/minicluster-data/ts-1/wal/instance:
uuid: "25ae7fefa24c400f8a5e752d3845fdea"
format_stamp: "Formatted at 2026-09-25 07:29:57 on dist-test-slave-57fn"
server_key: "39a7aecebfc8f41b566a0e2996dcd22d"
server_key_iv: "794b9128f462edf476773eb4abca5ec3"
server_key_version: "encryptionkey@0"
I20260925 07:29:57.689170 10804 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 07:29:57.689965 10819 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:29:57.690128 10804 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 07:29:57.690220 10804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790321392278483-9480-0/minicluster-data/ts-1/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790321392278483-9480-0/minicluster-data/ts-1/wal
uuid: "25ae7fefa24c400f8a5e752d3845fdea"
format_stamp: "Formatted at 2026-09-25 07:29:57 on dist-test-slave-57fn"
server_key: "39a7aecebfc8f41b566a0e2996dcd22d"
server_key_iv: "794b9128f462edf476773eb4abca5ec3"
server_key_version: "encryptionkey@0"
I20260925 07:29:57.690301 10804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790321392278483-9480-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790321392278483-9480-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790321392278483-9480-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:29:57.705642 10804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:29:57.707284 10804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:29:57.707442 10804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:29:57.707975 10804 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:29:57.708037 10804 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:29:57.708070 10804 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:29:57.708097 10804 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 07:29:57 dist-test-slave-57fn krb5kdc[10547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321397, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.2@KRBTEST.COM for kudu/127.9.66.62@KRBTEST.COM
I20260925 07:29:57.715065 10804 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.2:34311
I20260925 07:29:57.715189 10934 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.2:34311 every 8 connection(s)
I20260925 07:29:57.715446 10804 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790321392278483-9480-0/minicluster-data/ts-1/data/info.pb
I20260925 07:29:57.718291 10642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:29:57.708429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.2:47115 (local address 127.9.66.62:37099)
0925 07:29:57.708495 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 07:29:57.708497 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:29:57.708725 (+ 228us) server_negotiation.cc:407] Connection header received
0925 07:29:57.709160 (+ 435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:29:57.709161 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:29:57.709172 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:29:57.709247 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:29:57.709936 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:29:57.710758 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:29:57.712380 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:29:57.712689 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:29:57.712762 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:29:57.715482 (+ 2720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:29:57.715484 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:29:57.715484 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:29:57.715491 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:29:57.716928 (+ 1437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:29:57.717335 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:29:57.717337 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:29:57.717337 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:29:57.717406 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:29:57.717689 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:29:57.717690 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:29:57.717691 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:29:57.717803 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:29:57.717877 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 07:29:57.718130 (+ 253us) server_negotiation.cc:299] Negotiation successful
0925 07:29:57.718207 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 07:29:57.718441 10828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:29:57.708419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:37099 (local address 127.9.66.2:47115)
0925 07:29:57.708581 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 07:29:57.708619 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 07:29:57.709071 (+ 452us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:29:57.709322 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:29:57.709325 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:29:57.709658 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:29:57.709821 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:29:57.709825 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:29:57.710867 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:29:57.710868 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:29:57.712286 (+ 1418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:29:57.712289 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:29:57.712798 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:29:57.712799 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:29:57.712966 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:29:57.713446 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:29:57.713457 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:29:57.715387 (+ 1930us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:29:57.717030 (+ 1643us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:29:57.717036 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:29:57.717038 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:29:57.717246 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:29:57.717471 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:29:57.717472 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:29:57.717472 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:29:57.717621 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:29:57.717868 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:29:57.717874 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:29:57.718006 (+ 132us) client_negotiation.cc:769] Sending connection context
0925 07:29:57.718095 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 07:29:57.718233 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":101,"threads_started":1}
I20260925 07:29:57.718535 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 10804
I20260925 07:29:57.718629 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790321392278483-9480-0/minicluster-data/ts-1/wal/instance
I20260925 07:29:57.718799 9480 external_mini_cluster.cc:1428] Setting key 138d84e495e2de317c402403bcf6f807
I20260925 07:29:57.718858 10935 heartbeater.cc:347] Connected to a master server at 127.9.66.62:37099
I20260925 07:29:57.718935 10935 heartbeater.cc:464] Registering TS with master...
I20260925 07:29:57.719108 10935 heartbeater.cc:511] Master 127.9.66.62:37099 requested a full tablet report, sending...
I20260925 07:29:57.719473 10591 ts_manager.cc:194] Registered new tserver with Master: 25ae7fefa24c400f8a5e752d3845fdea (127.9.66.2:34311)
I20260925 07:29:57.719965 10591 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.66.2@KRBTEST.COM'} at 127.9.66.2:47115
I20260925 07:29:57.726579 10942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:29:57.720923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:37099 (local address 127.9.66.2:39859)
0925 07:29:57.721050 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0925 07:29:57.721061 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 07:29:57.721123 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:29:57.721237 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:29:57.721238 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:29:57.721444 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:29:57.721547 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:29:57.721549 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:29:57.722163 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:29:57.722164 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:29:57.722677 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:29:57.722679 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:29:57.723169 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:29:57.723170 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:29:57.723257 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:29:57.723640 (+ 383us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:29:57.723644 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:29:57.723931 (+ 287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:29:57.725630 (+ 1699us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:29:57.725630 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:29:57.725631 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:29:57.725782 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:29:57.725993 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:29:57.725994 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:29:57.725994 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:29:57.726040 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:29:57.726361 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:29:57.726361 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:29:57.726392 (+ 31us) client_negotiation.cc:769] Sending connection context
0925 07:29:57.726451 (+ 59us) client_negotiation.cc:240] Negotiation successful
0925 07:29:57.726517 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":80,"threads_started":1}
I20260925 07:29:57.726657 10642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:29:57.720961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.2:39859 (local address 127.9.66.62:37099)
0925 07:29:57.720999 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 07:29:57.721001 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:29:57.721124 (+ 123us) server_negotiation.cc:407] Connection header received
0925 07:29:57.721176 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:29:57.721177 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:29:57.721185 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:29:57.721224 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:29:57.721618 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:29:57.722055 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:29:57.722767 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:29:57.723087 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:29:57.723157 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:29:57.724010 (+ 853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:29:57.724012 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:29:57.724012 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:29:57.724018 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:29:57.725530 (+ 1512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:29:57.725853 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:29:57.725855 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:29:57.725855 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:29:57.725927 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:29:57.726096 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:29:57.726097 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:29:57.726097 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:29:57.726292 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:29:57.726359 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 07:29:57.726492 (+ 133us) server_negotiation.cc:299] Negotiation successful
0925 07:29:57.726563 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
WARNING: no policy specified for kudu/127.9.66.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.66.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.66.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.3.keytab.
Entry for principal kudu/127.9.66.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.3.keytab.
I20260925 07:29:57.745342 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790321392278483-9480-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790321392278483-9480-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790321392278483-9480-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790321392278483-9480-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.66.3:0
--local_ip_for_outbound_sockets=127.9.66.3
--webserver_interface=127.9.66.3
--webserver_port=0
--tserver_master_addrs=127.9.66.62:37099
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.3.keytab
--principal=kudu/127.9.66.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:33391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:29:57.825013 10944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:29:57.825189 10944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:29:57.825212 10944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:29:57.826372 10944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:29:57.826414 10944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:29:57.826433 10944 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 07:29:57.826450 10944 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.66.3
I20260925 07:29:57.827929 10944 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:33391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790321392278483-9480-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790321392278483-9480-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.3.keytab
--principal=kudu/127.9.66.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.66.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790321392278483-9480-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.66.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.66.62:37099
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10944
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.66.3
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790321392278483-9480-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:29:57.828094 10944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:29:57.828253 10944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 07:29:57.833413 10944 server_base.cc:1034] running on GCE node
W20260925 07:29:57.833344 10952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:29:57.833349 10949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:29:57.833379 10950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:29:57.833717 10944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:29:57.833899 10944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:29:57.835032 10944 hybrid_clock.cc:648] HybridClock initialized: now 1790321397835018 us; error 31 us; skew 500 ppm
Sep 25 07:29:57 dist-test-slave-57fn krb5kdc[10547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321397, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:29:57.837344 10944 init.cc:377] Logged in from keytab as kudu/127.9.66.3@KRBTEST.COM (short username kudu)
I20260925 07:29:57.837867 10944 webserver.cc:460] Webserver started at http://127.9.66.3:36381/ using document root <none> and password file <none>
I20260925 07:29:57.838116 10944 fs_manager.cc:359] Metadata directory not provided
I20260925 07:29:57.838181 10944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:29:57.838325 10944 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:29:57.839462 10944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790321392278483-9480-0/minicluster-data/ts-2/data/instance:
uuid: "a1ae730e8121439e940d5a61d67ddeff"
format_stamp: "Formatted at 2026-09-25 07:29:57 on dist-test-slave-57fn"
server_key: "836d16d932d641a8a5123a689eb12ba4"
server_key_iv: "6deeb303bd71cd456098b45bc19c2472"
server_key_version: "encryptionkey@0"
I20260925 07:29:57.839826 10944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790321392278483-9480-0/minicluster-data/ts-2/wal/instance:
uuid: "a1ae730e8121439e940d5a61d67ddeff"
format_stamp: "Formatted at 2026-09-25 07:29:57 on dist-test-slave-57fn"
server_key: "836d16d932d641a8a5123a689eb12ba4"
server_key_iv: "6deeb303bd71cd456098b45bc19c2472"
server_key_version: "encryptionkey@0"
I20260925 07:29:57.841091 10944 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 07:29:57.841830 10959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:29:57.842034 10944 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 07:29:57.842163 10944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790321392278483-9480-0/minicluster-data/ts-2/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790321392278483-9480-0/minicluster-data/ts-2/wal
uuid: "a1ae730e8121439e940d5a61d67ddeff"
format_stamp: "Formatted at 2026-09-25 07:29:57 on dist-test-slave-57fn"
server_key: "836d16d932d641a8a5123a689eb12ba4"
server_key_iv: "6deeb303bd71cd456098b45bc19c2472"
server_key_version: "encryptionkey@0"
I20260925 07:29:57.842247 10944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790321392278483-9480-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790321392278483-9480-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790321392278483-9480-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:29:57.853152 10944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:29:57.855290 10944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:29:57.855441 10944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:29:57.856002 10944 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:29:57.856067 10944 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:29:57.856104 10944 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:29:57.856122 10944 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 07:29:57 dist-test-slave-57fn krb5kdc[10547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321397, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.3@KRBTEST.COM for kudu/127.9.66.62@KRBTEST.COM
I20260925 07:29:57.862991 10944 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.3:40999
I20260925 07:29:57.863170 11074 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.3:40999 every 8 connection(s)
I20260925 07:29:57.863371 10944 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790321392278483-9480-0/minicluster-data/ts-2/data/info.pb
I20260925 07:29:57.865334 10642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:29:57.856358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.3:47853 (local address 127.9.66.62:37099)
0925 07:29:57.856417 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 07:29:57.856419 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:29:57.856631 (+ 212us) server_negotiation.cc:407] Connection header received
0925 07:29:57.857054 (+ 423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:29:57.857055 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:29:57.857066 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:29:57.857125 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:29:57.857983 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:29:57.858565 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:29:57.860145 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:29:57.860365 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:29:57.860416 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:29:57.862781 (+ 2365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:29:57.862783 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:29:57.862783 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:29:57.862791 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:29:57.864311 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:29:57.864639 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:29:57.864640 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:29:57.864640 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:29:57.864691 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:29:57.864882 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:29:57.864882 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:29:57.864883 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:29:57.864988 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:29:57.865037 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 07:29:57.865222 (+ 185us) server_negotiation.cc:299] Negotiation successful
0925 07:29:57.865277 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 07:29:57.865638 10968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:29:57.856368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:37099 (local address 127.9.66.3:47853)
0925 07:29:57.856516 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0925 07:29:57.856548 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0925 07:29:57.856953 (+ 405us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:29:57.857233 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:29:57.857237 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:29:57.857606 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:29:57.857880 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:29:57.857885 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:29:57.858654 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:29:57.858654 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:29:57.860038 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:29:57.860041 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:29:57.860432 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:29:57.860434 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:29:57.860595 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:29:57.861102 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:29:57.861111 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:29:57.862678 (+ 1567us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:29:57.864379 (+ 1701us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:29:57.864382 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:29:57.864384 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:29:57.864574 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:29:57.864744 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:29:57.864744 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:29:57.864744 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:29:57.864827 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:29:57.865041 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:29:57.865044 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:29:57.865139 (+ 95us) client_negotiation.cc:769] Sending connection context
0925 07:29:57.865210 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 07:29:57.865312 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":80,"threads_started":1}
I20260925 07:29:57.866112 11075 heartbeater.cc:347] Connected to a master server at 127.9.66.62:37099
I20260925 07:29:57.866189 11075 heartbeater.cc:464] Registering TS with master...
I20260925 07:29:57.866374 11075 heartbeater.cc:511] Master 127.9.66.62:37099 requested a full tablet report, sending...
I20260925 07:29:57.866760 10592 ts_manager.cc:194] Registered new tserver with Master: a1ae730e8121439e940d5a61d67ddeff (127.9.66.3:40999)
I20260925 07:29:57.867130 10592 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.66.3@KRBTEST.COM'} at 127.9.66.3:47853
I20260925 07:29:57.869686 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 10944
I20260925 07:29:57.869772 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790321392278483-9480-0/minicluster-data/ts-2/wal/instance
I20260925 07:29:57.869941 9480 external_mini_cluster.cc:1428] Setting key a9473cf318fc6b828f381042b49b018e
I20260925 07:29:57.870842 9480 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 07:29:57.873011 10642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:29:57.867207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.3:38577 (local address 127.9.66.62:37099)
0925 07:29:57.867259 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 07:29:57.867261 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:29:57.867502 (+ 241us) server_negotiation.cc:407] Connection header received
0925 07:29:57.867551 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:29:57.867552 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:29:57.867562 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:29:57.867618 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:29:57.867937 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:29:57.868477 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:29:57.869107 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:29:57.869267 (+ 160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:29:57.869311 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:29:57.870308 (+ 997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:29:57.870309 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:29:57.870310 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:29:57.870315 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:29:57.871934 (+ 1619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:29:57.872236 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:29:57.872237 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:29:57.872237 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:29:57.872292 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:29:57.872461 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:29:57.872462 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:29:57.872463 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:29:57.872609 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:29:57.872678 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 07:29:57.872861 (+ 183us) server_negotiation.cc:299] Negotiation successful
0925 07:29:57.872928 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 07:29:57.873028 11081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:29:57.867159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:37099 (local address 127.9.66.3:38577)
0925 07:29:57.867396 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0925 07:29:57.867406 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 07:29:57.867461 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:29:57.867631 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:29:57.867631 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:29:57.867773 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:29:57.867828 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:29:57.867829 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:29:57.868561 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:29:57.868562 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:29:57.869041 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:29:57.869042 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:29:57.869326 (+ 284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:29:57.869327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:29:57.869379 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:29:57.869797 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:29:57.869802 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:29:57.870187 (+ 385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:29:57.872014 (+ 1827us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:29:57.872014 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:29:57.872014 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:29:57.872168 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:29:57.872355 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:29:57.872356 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:29:57.872356 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:29:57.872405 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:29:57.872711 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:29:57.872711 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:29:57.872758 (+ 47us) client_negotiation.cc:769] Sending connection context
0925 07:29:57.872825 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 07:29:57.872929 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":60,"threads_started":1}
Sep 25 07:29:57 dist-test-slave-57fn krb5kdc[10547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321397, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 07:29:57 dist-test-slave-57fn krb5kdc[10547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321397, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.9.66.62@KRBTEST.COM
I20260925 07:29:57.892145 10642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:29:57.885985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46152 (local address 127.9.66.62:37099)
0925 07:29:57.886039 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 07:29:57.886041 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:29:57.886069 (+ 28us) server_negotiation.cc:407] Connection header received
0925 07:29:57.886086 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:29:57.886087 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:29:57.886094 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:29:57.886131 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:29:57.886706 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:29:57.887161 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:29:57.887848 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:29:57.888021 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:29:57.888066 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:29:57.890061 (+ 1995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:29:57.890062 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:29:57.890062 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:29:57.890068 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:29:57.891354 (+ 1286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:29:57.891632 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:29:57.891632 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:29:57.891632 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:29:57.891670 (+ 38us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:29:57.891802 (+ 132us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:29:57.891802 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:29:57.891803 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:29:57.891886 (+ 83us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:29:57.891947 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 07:29:57.892040 (+ 93us) server_negotiation.cc:299] Negotiation successful
0925 07:29:57.892090 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 07:29:57.893654 10592 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46152:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 07:29:57.895233 10592 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 07:29:57.901306 11092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:29:57.897193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.2:34311 (local address 127.0.0.1:56602)
0925 07:29:57.897338 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 07:29:57.897373 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 07:29:57.897624 (+ 251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:29:57.897821 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:29:57.897823 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:29:57.897825 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:29:57.897900 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:29:57.897901 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:29:57.899191 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:29:57.899192 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:29:57.900111 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:29:57.900113 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:29:57.901001 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:29:57.901002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:29:57.901091 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:29:57.901107 (+ 16us) client_negotiation.cc:769] Sending connection context
0925 07:29:57.901176 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 07:29:57.901232 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":52,"threads_started":1}
I20260925 07:29:57.901461 11094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:29:57.897221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.1:40653 (local address 127.0.0.1:48974)
0925 07:29:57.897905 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0925 07:29:57.897914 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 07:29:57.897958 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:29:57.898159 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:29:57.898160 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:29:57.898161 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:29:57.898194 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:29:57.898195 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:29:57.899357 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:29:57.899357 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:29:57.900456 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:29:57.900459 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:29:57.901226 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:29:57.901227 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:29:57.901281 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:29:57.901285 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 07:29:57.901330 (+ 45us) client_negotiation.cc:240] Negotiation successful
0925 07:29:57.901370 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":654,"thread_start_us":81,"threads_started":1}
I20260925 07:29:57.901335 11095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:29:57.897269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56602 (local address 127.9.66.2:34311)
0925 07:29:57.897431 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 07:29:57.897435 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 07:29:57.897665 (+ 230us) server_negotiation.cc:407] Connection header received
0925 07:29:57.897707 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:29:57.897709 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:29:57.897725 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:29:57.897806 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:29:57.897993 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:29:57.899089 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:29:57.900199 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:29:57.900912 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:29:57.900986 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:29:57.901038 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 07:29:57.901200 (+ 162us) server_negotiation.cc:299] Negotiation successful
0925 07:29:57.901238 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":70,"threads_started":1}
I20260925 07:29:57.901710 11096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:29:57.897444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48974 (local address 127.9.66.1:40653)
0925 07:29:57.897606 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 07:29:57.897610 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 07:29:57.898019 (+ 409us) server_negotiation.cc:407] Connection header received
0925 07:29:57.898064 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:29:57.898066 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:29:57.898081 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:29:57.898163 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:29:57.898247 (+ 84us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:29:57.899271 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:29:57.900541 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:29:57.901067 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:29:57.901138 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:29:57.901195 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 07:29:57.901573 (+ 378us) server_negotiation.cc:299] Negotiation successful
0925 07:29:57.901617 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":119,"threads_started":1}
I20260925 07:29:57.902374 10869 tablet_service.cc:1467] Processing CreateTablet for tablet f575e6c6766c4d71aaba804fdd265f1c (DEFAULT_TABLE table=test-table [id=389078d4503c4f0bb76b356a23629283]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:29:57.902388 10729 tablet_service.cc:1467] Processing CreateTablet for tablet f575e6c6766c4d71aaba804fdd265f1c (DEFAULT_TABLE table=test-table [id=389078d4503c4f0bb76b356a23629283]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:29:57.902680 11093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:29:57.897221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.3:40999 (local address 127.0.0.1:53860)
0925 07:29:57.897476 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0925 07:29:57.897487 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 07:29:57.897580 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:29:57.897871 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:29:57.897872 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:29:57.897873 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:29:57.897905 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:29:57.897906 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:29:57.900341 (+ 2435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:29:57.900342 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:29:57.901375 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:29:57.901377 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:29:57.902453 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:29:57.902454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:29:57.902514 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:29:57.902517 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 07:29:57.902578 (+ 61us) client_negotiation.cc:240] Negotiation successful
0925 07:29:57.902612 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":79,"threads_started":1}
I20260925 07:29:57.902731 11097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:29:57.897438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53860 (local address 127.9.66.3:40999)
0925 07:29:57.897721 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0925 07:29:57.897724 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:29:57.897732 (+ 8us) server_negotiation.cc:407] Connection header received
0925 07:29:57.897764 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:29:57.897766 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:29:57.897779 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:29:57.897856 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:29:57.899209 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:29:57.900261 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:29:57.901480 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:29:57.902279 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:29:57.902364 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:29:57.902416 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 07:29:57.902594 (+ 178us) server_negotiation.cc:299] Negotiation successful
0925 07:29:57.902635 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":98,"threads_started":1}
I20260925 07:29:57.902743 10869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f575e6c6766c4d71aaba804fdd265f1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:29:57.902743 10729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f575e6c6766c4d71aaba804fdd265f1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:29:57.905303 11098 tablet_bootstrap.cc:492] T f575e6c6766c4d71aaba804fdd265f1c P 724be09c3ade4d0d9043cce4e15a7e26: Bootstrap starting.
I20260925 07:29:57.905535 11099 tablet_bootstrap.cc:492] T f575e6c6766c4d71aaba804fdd265f1c P 25ae7fefa24c400f8a5e752d3845fdea: Bootstrap starting.
I20260925 07:29:57.905947 11009 tablet_service.cc:1467] Processing CreateTablet for tablet f575e6c6766c4d71aaba804fdd265f1c (DEFAULT_TABLE table=test-table [id=389078d4503c4f0bb76b356a23629283]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:29:57.906052 11099 tablet_bootstrap.cc:654] T f575e6c6766c4d71aaba804fdd265f1c P 25ae7fefa24c400f8a5e752d3845fdea: Neither blocks nor log segments found. Creating new log.
I20260925 07:29:57.906229 11009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f575e6c6766c4d71aaba804fdd265f1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:29:57.906369 11099 log.cc:824] T f575e6c6766c4d71aaba804fdd265f1c P 25ae7fefa24c400f8a5e752d3845fdea: Log is configured to *not* fsync() on all Append() calls
I20260925 07:29:57.906996 11099 tablet_bootstrap.cc:492] T f575e6c6766c4d71aaba804fdd265f1c P 25ae7fefa24c400f8a5e752d3845fdea: No bootstrap required, opened a new log
I20260925 07:29:57.907045 11099 ts_tablet_manager.cc:1397] T f575e6c6766c4d71aaba804fdd265f1c P 25ae7fefa24c400f8a5e752d3845fdea: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 07:29:57.908510 11101 tablet_bootstrap.cc:492] T f575e6c6766c4d71aaba804fdd265f1c P a1ae730e8121439e940d5a61d67ddeff: Bootstrap starting.
I20260925 07:29:57.908495 11099 raft_consensus.cc:348] T f575e6c6766c4d71aaba804fdd265f1c P 25ae7fefa24c400f8a5e752d3845fdea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25ae7fefa24c400f8a5e752d3845fdea" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 34311 } } peers { permanent_uuid: "724be09c3ade4d0d9043cce4e15a7e26" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 40653 } } peers { permanent_uuid: "a1ae730e8121439e940d5a61d67ddeff" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 40999 } }
I20260925 07:29:57.908808 11099 raft_consensus.cc:374] T f575e6c6766c4d71aaba804fdd265f1c P 25ae7fefa24c400f8a5e752d3845fdea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:29:57.908878 11099 raft_consensus.cc:729] T f575e6c6766c4d71aaba804fdd265f1c P 25ae7fefa24c400f8a5e752d3845fdea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25ae7fefa24c400f8a5e752d3845fdea, State: Initialized, Role: FOLLOWER
I20260925 07:29:57.909024 11098 tablet_bootstrap.cc:654] T f575e6c6766c4d71aaba804fdd265f1c P 724be09c3ade4d0d9043cce4e15a7e26: Neither blocks nor log segments found. Creating new log.
I20260925 07:29:57.909047 11099 consensus_queue.cc:260] T f575e6c6766c4d71aaba804fdd265f1c P 25ae7fefa24c400f8a5e752d3845fdea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25ae7fefa24c400f8a5e752d3845fdea" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 34311 } } peers { permanent_uuid: "724be09c3ade4d0d9043cce4e15a7e26" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 40653 } } peers { permanent_uuid: "a1ae730e8121439e940d5a61d67ddeff" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 40999 } }
I20260925 07:29:57.909340 10935 heartbeater.cc:503] Master 127.9.66.62:37099 was elected leader, sending a full tablet report...
I20260925 07:29:57.909365 11098 log.cc:824] T f575e6c6766c4d71aaba804fdd265f1c P 724be09c3ade4d0d9043cce4e15a7e26: Log is configured to *not* fsync() on all Append() calls
I20260925 07:29:57.909619 11099 ts_tablet_manager.cc:1428] T f575e6c6766c4d71aaba804fdd265f1c P 25ae7fefa24c400f8a5e752d3845fdea: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260925 07:29:57.909631 11101 tablet_bootstrap.cc:654] T f575e6c6766c4d71aaba804fdd265f1c P a1ae730e8121439e940d5a61d67ddeff: Neither blocks nor log segments found. Creating new log.
I20260925 07:29:57.909924 11101 log.cc:824] T f575e6c6766c4d71aaba804fdd265f1c P a1ae730e8121439e940d5a61d67ddeff: Log is configured to *not* fsync() on all Append() calls
I20260925 07:29:57.910463 11098 tablet_bootstrap.cc:492] T f575e6c6766c4d71aaba804fdd265f1c P 724be09c3ade4d0d9043cce4e15a7e26: No bootstrap required, opened a new log
I20260925 07:29:57.910538 11098 ts_tablet_manager.cc:1397] T f575e6c6766c4d71aaba804fdd265f1c P 724be09c3ade4d0d9043cce4e15a7e26: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260925 07:29:57.910678 11101 tablet_bootstrap.cc:492] T f575e6c6766c4d71aaba804fdd265f1c P a1ae730e8121439e940d5a61d67ddeff: No bootstrap required, opened a new log
I20260925 07:29:57.910749 11101 ts_tablet_manager.cc:1397] T f575e6c6766c4d71aaba804fdd265f1c P a1ae730e8121439e940d5a61d67ddeff: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 07:29:57.911921 11098 raft_consensus.cc:348] T f575e6c6766c4d71aaba804fdd265f1c P 724be09c3ade4d0d9043cce4e15a7e26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25ae7fefa24c400f8a5e752d3845fdea" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 34311 } } peers { permanent_uuid: "724be09c3ade4d0d9043cce4e15a7e26" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 40653 } } peers { permanent_uuid: "a1ae730e8121439e940d5a61d67ddeff" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 40999 } }
I20260925 07:29:57.912052 11098 raft_consensus.cc:374] T f575e6c6766c4d71aaba804fdd265f1c P 724be09c3ade4d0d9043cce4e15a7e26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:29:57.912074 11098 raft_consensus.cc:729] T f575e6c6766c4d71aaba804fdd265f1c P 724be09c3ade4d0d9043cce4e15a7e26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 724be09c3ade4d0d9043cce4e15a7e26, State: Initialized, Role: FOLLOWER
I20260925 07:29:57.912035 11101 raft_consensus.cc:348] T f575e6c6766c4d71aaba804fdd265f1c P a1ae730e8121439e940d5a61d67ddeff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25ae7fefa24c400f8a5e752d3845fdea" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 34311 } } peers { permanent_uuid: "724be09c3ade4d0d9043cce4e15a7e26" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 40653 } } peers { permanent_uuid: "a1ae730e8121439e940d5a61d67ddeff" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 40999 } }
I20260925 07:29:57.912187 11098 consensus_queue.cc:260] T f575e6c6766c4d71aaba804fdd265f1c P 724be09c3ade4d0d9043cce4e15a7e26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25ae7fefa24c400f8a5e752d3845fdea" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 34311 } } peers { permanent_uuid: "724be09c3ade4d0d9043cce4e15a7e26" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 40653 } } peers { permanent_uuid: "a1ae730e8121439e940d5a61d67ddeff" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 40999 } }
I20260925 07:29:57.912214 11101 raft_consensus.cc:374] T f575e6c6766c4d71aaba804fdd265f1c P a1ae730e8121439e940d5a61d67ddeff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:29:57.912256 11101 raft_consensus.cc:729] T f575e6c6766c4d71aaba804fdd265f1c P a1ae730e8121439e940d5a61d67ddeff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1ae730e8121439e940d5a61d67ddeff, State: Initialized, Role: FOLLOWER
I20260925 07:29:57.912362 11101 consensus_queue.cc:260] T f575e6c6766c4d71aaba804fdd265f1c P a1ae730e8121439e940d5a61d67ddeff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25ae7fefa24c400f8a5e752d3845fdea" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 34311 } } peers { permanent_uuid: "724be09c3ade4d0d9043cce4e15a7e26" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 40653 } } peers { permanent_uuid: "a1ae730e8121439e940d5a61d67ddeff" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 40999 } }
I20260925 07:29:57.912442 11098 ts_tablet_manager.cc:1428] T f575e6c6766c4d71aaba804fdd265f1c P 724be09c3ade4d0d9043cce4e15a7e26: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 07:29:57.912482 10795 heartbeater.cc:503] Master 127.9.66.62:37099 was elected leader, sending a full tablet report...
I20260925 07:29:57.912652 11101 ts_tablet_manager.cc:1428] T f575e6c6766c4d71aaba804fdd265f1c P a1ae730e8121439e940d5a61d67ddeff: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260925 07:29:57.912688 11075 heartbeater.cc:503] Master 127.9.66.62:37099 was elected leader, sending a full tablet report...
I20260925 07:29:57.928318 11102 raft_consensus.cc:482] T f575e6c6766c4d71aaba804fdd265f1c P 25ae7fefa24c400f8a5e752d3845fdea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 07:29:57.928440 11102 raft_consensus.cc:504] T f575e6c6766c4d71aaba804fdd265f1c P 25ae7fefa24c400f8a5e752d3845fdea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25ae7fefa24c400f8a5e752d3845fdea" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 34311 } } peers { permanent_uuid: "724be09c3ade4d0d9043cce4e15a7e26" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 40653 } } peers { permanent_uuid: "a1ae730e8121439e940d5a61d67ddeff" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 40999 } }
I20260925 07:29:57.928850 11102 leader_election.cc:288] T f575e6c6766c4d71aaba804fdd265f1c P 25ae7fefa24c400f8a5e752d3845fdea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 724be09c3ade4d0d9043cce4e15a7e26 (127.9.66.1:40653), a1ae730e8121439e940d5a61d67ddeff (127.9.66.3:40999)
I20260925 07:29:57.932516 11096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:29:57.929769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.2:55903 (local address 127.9.66.1:40653)
0925 07:29:57.929809 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 07:29:57.929811 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:29:57.929818 (+ 7us) server_negotiation.cc:407] Connection header received
0925 07:29:57.929840 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:29:57.929841 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:29:57.929849 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:29:57.929898 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:29:57.930062 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:29:57.930779 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:29:57.931790 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:29:57.932147 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:29:57.932227 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:29:57.932237 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 07:29:57.932414 (+ 177us) server_negotiation.cc:299] Negotiation successful
0925 07:29:57.932453 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 07:29:57.932542 11107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:29:57.929088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.1:40653 (local address 127.9.66.2:55903)
0925 07:29:57.929470 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0925 07:29:57.929482 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 07:29:57.929571 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:29:57.929909 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:29:57.929910 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:29:57.929912 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:29:57.929989 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:29:57.929990 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:29:57.930881 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:29:57.930882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:29:57.931690 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:29:57.931692 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:29:57.932228 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:29:57.932229 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:29:57.932298 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:29:57.932301 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 07:29:57.932405 (+ 104us) client_negotiation.cc:240] Negotiation successful
0925 07:29:57.932443 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362,"mutex_wait_us":192,"thread_start_us":102,"threads_started":1}
I20260925 07:29:57.932803 10828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:29:57.929087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.3:40999 (local address 127.9.66.2:37783)
0925 07:29:57.929447 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0925 07:29:57.929481 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 07:29:57.929553 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:29:57.929701 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:29:57.929702 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:29:57.929704 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:29:57.929779 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:29:57.929781 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:29:57.930709 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:29:57.930710 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:29:57.931820 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:29:57.931822 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:29:57.932524 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:29:57.932525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:29:57.932629 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:29:57.932634 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 07:29:57.932708 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 07:29:57.932743 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"mutex_wait_us":34}
I20260925 07:29:57.932830 11097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:29:57.929255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.2:37783 (local address 127.9.66.3:40999)
0925 07:29:57.929307 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 07:29:57.929310 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:29:57.929609 (+ 299us) server_negotiation.cc:407] Connection header received
0925 07:29:57.929635 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:29:57.929636 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:29:57.929647 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:29:57.929694 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:29:57.929847 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:29:57.930629 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:29:57.931926 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:29:57.932444 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:29:57.932525 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:29:57.932535 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 07:29:57.932738 (+ 203us) server_negotiation.cc:299] Negotiation successful
0925 07:29:57.932779 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 07:29:57.932968 10749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f575e6c6766c4d71aaba804fdd265f1c" candidate_uuid: "25ae7fefa24c400f8a5e752d3845fdea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "724be09c3ade4d0d9043cce4e15a7e26" is_pre_election: true
I20260925 07:29:57.933144 10749 raft_consensus.cc:2454] T f575e6c6766c4d71aaba804fdd265f1c P 724be09c3ade4d0d9043cce4e15a7e26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 25ae7fefa24c400f8a5e752d3845fdea in term 0.
I20260925 07:29:57.933110 11029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f575e6c6766c4d71aaba804fdd265f1c" candidate_uuid: "25ae7fefa24c400f8a5e752d3845fdea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1ae730e8121439e940d5a61d67ddeff" is_pre_election: true
I20260925 07:29:57.933274 11029 raft_consensus.cc:2454] T f575e6c6766c4d71aaba804fdd265f1c P a1ae730e8121439e940d5a61d67ddeff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 25ae7fefa24c400f8a5e752d3845fdea in term 0.
I20260925 07:29:57.933395 10823 leader_election.cc:302] T f575e6c6766c4d71aaba804fdd265f1c P 25ae7fefa24c400f8a5e752d3845fdea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25ae7fefa24c400f8a5e752d3845fdea, 724be09c3ade4d0d9043cce4e15a7e26; no voters:
I20260925 07:29:57.933575 11102 raft_consensus.cc:2781] T f575e6c6766c4d71aaba804fdd265f1c P 25ae7fefa24c400f8a5e752d3845fdea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 07:29:57.933694 11102 raft_consensus.cc:482] T f575e6c6766c4d71aaba804fdd265f1c P 25ae7fefa24c400f8a5e752d3845fdea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 07:29:57.933725 11102 raft_consensus.cc:3037] T f575e6c6766c4d71aaba804fdd265f1c P 25ae7fefa24c400f8a5e752d3845fdea [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:29:57.934602 11102 raft_consensus.cc:504] T f575e6c6766c4d71aaba804fdd265f1c P 25ae7fefa24c400f8a5e752d3845fdea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25ae7fefa24c400f8a5e752d3845fdea" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 34311 } } peers { permanent_uuid: "724be09c3ade4d0d9043cce4e15a7e26" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 40653 } } peers { permanent_uuid: "a1ae730e8121439e940d5a61d67ddeff" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 40999 } }
I20260925 07:29:57.934743 11102 leader_election.cc:288] T f575e6c6766c4d71aaba804fdd265f1c P 25ae7fefa24c400f8a5e752d3845fdea [CANDIDATE]: Term 1 election: Requested vote from peers 724be09c3ade4d0d9043cce4e15a7e26 (127.9.66.1:40653), a1ae730e8121439e940d5a61d67ddeff (127.9.66.3:40999)
I20260925 07:29:57.934880 11029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f575e6c6766c4d71aaba804fdd265f1c" candidate_uuid: "25ae7fefa24c400f8a5e752d3845fdea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1ae730e8121439e940d5a61d67ddeff"
I20260925 07:29:57.934896 10749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f575e6c6766c4d71aaba804fdd265f1c" candidate_uuid: "25ae7fefa24c400f8a5e752d3845fdea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "724be09c3ade4d0d9043cce4e15a7e26"
I20260925 07:29:57.934984 11029 raft_consensus.cc:3037] T f575e6c6766c4d71aaba804fdd265f1c P a1ae730e8121439e940d5a61d67ddeff [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:29:57.935024 10749 raft_consensus.cc:3037] T f575e6c6766c4d71aaba804fdd265f1c P 724be09c3ade4d0d9043cce4e15a7e26 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:29:57.935760 11029 raft_consensus.cc:2454] T f575e6c6766c4d71aaba804fdd265f1c P a1ae730e8121439e940d5a61d67ddeff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25ae7fefa24c400f8a5e752d3845fdea in term 1.
I20260925 07:29:57.935761 10749 raft_consensus.cc:2454] T f575e6c6766c4d71aaba804fdd265f1c P 724be09c3ade4d0d9043cce4e15a7e26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25ae7fefa24c400f8a5e752d3845fdea in term 1.
I20260925 07:29:57.935921 10821 leader_election.cc:302] T f575e6c6766c4d71aaba804fdd265f1c P 25ae7fefa24c400f8a5e752d3845fdea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25ae7fefa24c400f8a5e752d3845fdea, a1ae730e8121439e940d5a61d67ddeff; no voters:
I20260925 07:29:57.936036 11102 raft_consensus.cc:2781] T f575e6c6766c4d71aaba804fdd265f1c P 25ae7fefa24c400f8a5e752d3845fdea [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:29:57.936159 11102 raft_consensus.cc:686] T f575e6c6766c4d71aaba804fdd265f1c P 25ae7fefa24c400f8a5e752d3845fdea [term 1 LEADER]: Becoming Leader. State: Replica: 25ae7fefa24c400f8a5e752d3845fdea, State: Running, Role: LEADER
I20260925 07:29:57.936246 11102 consensus_queue.cc:237] T f575e6c6766c4d71aaba804fdd265f1c P 25ae7fefa24c400f8a5e752d3845fdea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25ae7fefa24c400f8a5e752d3845fdea" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 34311 } } peers { permanent_uuid: "724be09c3ade4d0d9043cce4e15a7e26" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 40653 } } peers { permanent_uuid: "a1ae730e8121439e940d5a61d67ddeff" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 40999 } }
I20260925 07:29:57.937027 10591 catalog_manager.cc:5495] T f575e6c6766c4d71aaba804fdd265f1c P 25ae7fefa24c400f8a5e752d3845fdea reported cstate change: term changed from 0 to 1, leader changed from <none> to 25ae7fefa24c400f8a5e752d3845fdea (127.9.66.2). New cstate: current_term: 1 leader_uuid: "25ae7fefa24c400f8a5e752d3845fdea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25ae7fefa24c400f8a5e752d3845fdea" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 34311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "724be09c3ade4d0d9043cce4e15a7e26" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 40653 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1ae730e8121439e940d5a61d67ddeff" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 40999 } health_report { overall_health: UNKNOWN } } }
I20260925 07:29:57.951185 11095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:29:57.949046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56612 (local address 127.9.66.2:34311)
0925 07:29:57.949110 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 07:29:57.949112 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:29:57.949119 (+ 7us) server_negotiation.cc:407] Connection header received
0925 07:29:57.949196 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:29:57.949197 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:29:57.949211 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:29:57.949269 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 07:29:57.949385 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:29:57.949856 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:29:57.950586 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:29:57.950756 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:29:57.950802 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:29:57.950953 (+ 151us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 07:29:57.951034 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 07:29:57.951084 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 07:29:57.951114 (+ 30us) server_negotiation.cc:299] Negotiation successful
0925 07:29:57.951128 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 07:29:57.952664 11029 raft_consensus.cc:1261] T f575e6c6766c4d71aaba804fdd265f1c P a1ae730e8121439e940d5a61d67ddeff [term 1 FOLLOWER]: Refusing update from remote peer 25ae7fefa24c400f8a5e752d3845fdea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:29:57.952672 10749 raft_consensus.cc:1261] T f575e6c6766c4d71aaba804fdd265f1c P 724be09c3ade4d0d9043cce4e15a7e26 [term 1 FOLLOWER]: Refusing update from remote peer 25ae7fefa24c400f8a5e752d3845fdea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:29:57.952957 11108 consensus_queue.cc:1035] T f575e6c6766c4d71aaba804fdd265f1c P 25ae7fefa24c400f8a5e752d3845fdea [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1ae730e8121439e940d5a61d67ddeff" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 40999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:29:57.953084 11102 consensus_queue.cc:1035] T f575e6c6766c4d71aaba804fdd265f1c P 25ae7fefa24c400f8a5e752d3845fdea [LEADER]: Connected to new peer: Peer: permanent_uuid: "724be09c3ade4d0d9043cce4e15a7e26" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 40653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:29:57.955091 11111 mvcc.cc:205] Tried to move back new op lower bound from 7333156446011850752 to 7333156445947576320. Current Snapshot: MvccSnapshot[applied={T|T < 7333156446011850752}]
I20260925 07:29:57.955423 11113 mvcc.cc:205] Tried to move back new op lower bound from 7333156446011850752 to 7333156445947576320. Current Snapshot: MvccSnapshot[applied={T|T < 7333156446011850752}]
I20260925 07:29:57.958125 10591 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46152:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 07:29:57.958197 10591 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46152:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 07:29:57.958890 10591 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 42650a67401745e9be59eb4711a81af7: Sending DeleteTablet for 3 replicas of tablet f575e6c6766c4d71aaba804fdd265f1c
I20260925 07:29:57.959192 10729 tablet_service.cc:1514] Processing DeleteTablet for tablet f575e6c6766c4d71aaba804fdd265f1c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:29:57 UTC) from {username='kudu', principal='kudu/127.9.66.62@KRBTEST.COM'} at 127.0.0.1:48974
I20260925 07:29:57.959190 10869 tablet_service.cc:1514] Processing DeleteTablet for tablet f575e6c6766c4d71aaba804fdd265f1c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:29:57 UTC) from {username='kudu', principal='kudu/127.9.66.62@KRBTEST.COM'} at 127.0.0.1:56602
I20260925 07:29:57.959192 11009 tablet_service.cc:1514] Processing DeleteTablet for tablet f575e6c6766c4d71aaba804fdd265f1c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:29:57 UTC) from {username='kudu', principal='kudu/127.9.66.62@KRBTEST.COM'} at 127.0.0.1:53860
I20260925 07:29:57.959388 11122 tablet_replica.cc:330] T f575e6c6766c4d71aaba804fdd265f1c P 724be09c3ade4d0d9043cce4e15a7e26: stopping tablet replica
I20260925 07:29:57.959486 11122 raft_consensus.cc:2229] T f575e6c6766c4d71aaba804fdd265f1c P 724be09c3ade4d0d9043cce4e15a7e26 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 07:29:57.959586 11120 tablet_replica.cc:330] T f575e6c6766c4d71aaba804fdd265f1c P a1ae730e8121439e940d5a61d67ddeff: stopping tablet replica
I20260925 07:29:57.959595 11122 raft_consensus.cc:2258] T f575e6c6766c4d71aaba804fdd265f1c P 724be09c3ade4d0d9043cce4e15a7e26 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:29:57.959667 11120 raft_consensus.cc:2229] T f575e6c6766c4d71aaba804fdd265f1c P a1ae730e8121439e940d5a61d67ddeff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 07:29:57.959761 11120 raft_consensus.cc:2258] T f575e6c6766c4d71aaba804fdd265f1c P a1ae730e8121439e940d5a61d67ddeff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:29:57.959713 10591 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46152:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 07:29:57.959839 10591 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 07:29:57.959923 11121 tablet_replica.cc:330] T f575e6c6766c4d71aaba804fdd265f1c P 25ae7fefa24c400f8a5e752d3845fdea: stopping tablet replica
I20260925 07:29:57.960003 11121 raft_consensus.cc:2229] T f575e6c6766c4d71aaba804fdd265f1c P 25ae7fefa24c400f8a5e752d3845fdea [term 1 LEADER]: Raft consensus shutting down.
I20260925 07:29:57.960114 11121 raft_consensus.cc:2258] T f575e6c6766c4d71aaba804fdd265f1c P 25ae7fefa24c400f8a5e752d3845fdea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:29:57.960427 11120 ts_tablet_manager.cc:1905] T f575e6c6766c4d71aaba804fdd265f1c P a1ae730e8121439e940d5a61d67ddeff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 07:29:57.960526 11121 ts_tablet_manager.cc:1905] T f575e6c6766c4d71aaba804fdd265f1c P 25ae7fefa24c400f8a5e752d3845fdea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 07:29:57.961272 10729 tablet_service.cc:1467] Processing CreateTablet for tablet 5c4d234392284a09b09d116964d4bc29 (DEFAULT_TABLE table=test-table [id=5eb3bfcabbeb4e91af451af20ac0ee63]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:29:57.961395 10729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c4d234392284a09b09d116964d4bc29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:29:57.962402 11121 ts_tablet_manager.cc:1918] T f575e6c6766c4d71aaba804fdd265f1c P 25ae7fefa24c400f8a5e752d3845fdea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 07:29:57.962481 11121 log.cc:1196] T f575e6c6766c4d71aaba804fdd265f1c P 25ae7fefa24c400f8a5e752d3845fdea: Deleting WAL directory at /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790321392278483-9480-0/minicluster-data/ts-1/wal/wals/f575e6c6766c4d71aaba804fdd265f1c
I20260925 07:29:57.962659 11120 ts_tablet_manager.cc:1918] T f575e6c6766c4d71aaba804fdd265f1c P a1ae730e8121439e940d5a61d67ddeff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 07:29:57.962729 11121 ts_tablet_manager.cc:1939] T f575e6c6766c4d71aaba804fdd265f1c P 25ae7fefa24c400f8a5e752d3845fdea: Deleting consensus metadata
I20260925 07:29:57.962818 11120 log.cc:1196] T f575e6c6766c4d71aaba804fdd265f1c P a1ae730e8121439e940d5a61d67ddeff: Deleting WAL directory at /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790321392278483-9480-0/minicluster-data/ts-2/wal/wals/f575e6c6766c4d71aaba804fdd265f1c
I20260925 07:29:57.963106 11009 tablet_service.cc:1467] Processing CreateTablet for tablet 5c4d234392284a09b09d116964d4bc29 (DEFAULT_TABLE table=test-table [id=5eb3bfcabbeb4e91af451af20ac0ee63]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:29:57.963135 11120 ts_tablet_manager.cc:1939] T f575e6c6766c4d71aaba804fdd265f1c P a1ae730e8121439e940d5a61d67ddeff: Deleting consensus metadata
I20260925 07:29:57.963248 11009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c4d234392284a09b09d116964d4bc29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:29:57.963541 10577 catalog_manager.cc:4841] TS a1ae730e8121439e940d5a61d67ddeff (127.9.66.3:40999): tablet f575e6c6766c4d71aaba804fdd265f1c (table test-table [id=389078d4503c4f0bb76b356a23629283]) successfully deleted
I20260925 07:29:57.964129 11101 tablet_bootstrap.cc:492] T 5c4d234392284a09b09d116964d4bc29 P a1ae730e8121439e940d5a61d67ddeff: Bootstrap starting.
I20260925 07:29:57.964162 11098 tablet_bootstrap.cc:492] T 5c4d234392284a09b09d116964d4bc29 P 724be09c3ade4d0d9043cce4e15a7e26: Bootstrap starting.
I20260925 07:29:57.964545 11101 tablet_bootstrap.cc:654] T 5c4d234392284a09b09d116964d4bc29 P a1ae730e8121439e940d5a61d67ddeff: Neither blocks nor log segments found. Creating new log.
I20260925 07:29:57.964545 11098 tablet_bootstrap.cc:654] T 5c4d234392284a09b09d116964d4bc29 P 724be09c3ade4d0d9043cce4e15a7e26: Neither blocks nor log segments found. Creating new log.
I20260925 07:29:57.965031 10576 catalog_manager.cc:4841] TS 25ae7fefa24c400f8a5e752d3845fdea (127.9.66.2:34311): tablet f575e6c6766c4d71aaba804fdd265f1c (table test-table [id=389078d4503c4f0bb76b356a23629283]) successfully deleted
I20260925 07:29:57.965116 10869 tablet_service.cc:1467] Processing CreateTablet for tablet 5c4d234392284a09b09d116964d4bc29 (DEFAULT_TABLE table=test-table [id=5eb3bfcabbeb4e91af451af20ac0ee63]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:29:57.965176 11098 tablet_bootstrap.cc:492] T 5c4d234392284a09b09d116964d4bc29 P 724be09c3ade4d0d9043cce4e15a7e26: No bootstrap required, opened a new log
I20260925 07:29:57.965210 10869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c4d234392284a09b09d116964d4bc29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:29:57.965217 11098 ts_tablet_manager.cc:1397] T 5c4d234392284a09b09d116964d4bc29 P 724be09c3ade4d0d9043cce4e15a7e26: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 07:29:57.965351 11098 raft_consensus.cc:348] T 5c4d234392284a09b09d116964d4bc29 P 724be09c3ade4d0d9043cce4e15a7e26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "724be09c3ade4d0d9043cce4e15a7e26" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 40653 } } peers { permanent_uuid: "25ae7fefa24c400f8a5e752d3845fdea" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 34311 } } peers { permanent_uuid: "a1ae730e8121439e940d5a61d67ddeff" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 40999 } }
I20260925 07:29:57.965416 11098 raft_consensus.cc:374] T 5c4d234392284a09b09d116964d4bc29 P 724be09c3ade4d0d9043cce4e15a7e26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:29:57.965436 11098 raft_consensus.cc:729] T 5c4d234392284a09b09d116964d4bc29 P 724be09c3ade4d0d9043cce4e15a7e26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 724be09c3ade4d0d9043cce4e15a7e26, State: Initialized, Role: FOLLOWER
I20260925 07:29:57.965480 11098 consensus_queue.cc:260] T 5c4d234392284a09b09d116964d4bc29 P 724be09c3ade4d0d9043cce4e15a7e26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "724be09c3ade4d0d9043cce4e15a7e26" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 40653 } } peers { permanent_uuid: "25ae7fefa24c400f8a5e752d3845fdea" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 34311 } } peers { permanent_uuid: "a1ae730e8121439e940d5a61d67ddeff" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 40999 } }
I20260925 07:29:57.965598 11098 ts_tablet_manager.cc:1428] T 5c4d234392284a09b09d116964d4bc29 P 724be09c3ade4d0d9043cce4e15a7e26: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 07:29:57.965651 11122 ts_tablet_manager.cc:1905] T f575e6c6766c4d71aaba804fdd265f1c P 724be09c3ade4d0d9043cce4e15a7e26: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 07:29:57.966511 11099 tablet_bootstrap.cc:492] T 5c4d234392284a09b09d116964d4bc29 P 25ae7fefa24c400f8a5e752d3845fdea: Bootstrap starting.
I20260925 07:29:57.966907 11099 tablet_bootstrap.cc:654] T 5c4d234392284a09b09d116964d4bc29 P 25ae7fefa24c400f8a5e752d3845fdea: Neither blocks nor log segments found. Creating new log.
I20260925 07:29:57.967260 11122 ts_tablet_manager.cc:1918] T f575e6c6766c4d71aaba804fdd265f1c P 724be09c3ade4d0d9043cce4e15a7e26: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 07:29:57.967329 11122 log.cc:1196] T f575e6c6766c4d71aaba804fdd265f1c P 724be09c3ade4d0d9043cce4e15a7e26: Deleting WAL directory at /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790321392278483-9480-0/minicluster-data/ts-0/wal/wals/f575e6c6766c4d71aaba804fdd265f1c
I20260925 07:29:57.967495 11099 tablet_bootstrap.cc:492] T 5c4d234392284a09b09d116964d4bc29 P 25ae7fefa24c400f8a5e752d3845fdea: No bootstrap required, opened a new log
I20260925 07:29:57.967557 11099 ts_tablet_manager.cc:1397] T 5c4d234392284a09b09d116964d4bc29 P 25ae7fefa24c400f8a5e752d3845fdea: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 07:29:57.967618 11122 ts_tablet_manager.cc:1939] T f575e6c6766c4d71aaba804fdd265f1c P 724be09c3ade4d0d9043cce4e15a7e26: Deleting consensus metadata
I20260925 07:29:57.967713 11099 raft_consensus.cc:348] T 5c4d234392284a09b09d116964d4bc29 P 25ae7fefa24c400f8a5e752d3845fdea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "724be09c3ade4d0d9043cce4e15a7e26" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 40653 } } peers { permanent_uuid: "25ae7fefa24c400f8a5e752d3845fdea" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 34311 } } peers { permanent_uuid: "a1ae730e8121439e940d5a61d67ddeff" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 40999 } }
I20260925 07:29:57.967767 11101 tablet_bootstrap.cc:492] T 5c4d234392284a09b09d116964d4bc29 P a1ae730e8121439e940d5a61d67ddeff: No bootstrap required, opened a new log
I20260925 07:29:57.967779 11099 raft_consensus.cc:374] T 5c4d234392284a09b09d116964d4bc29 P 25ae7fefa24c400f8a5e752d3845fdea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:29:57.967800 11099 raft_consensus.cc:729] T 5c4d234392284a09b09d116964d4bc29 P 25ae7fefa24c400f8a5e752d3845fdea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25ae7fefa24c400f8a5e752d3845fdea, State: Initialized, Role: FOLLOWER
I20260925 07:29:57.967805 11101 ts_tablet_manager.cc:1397] T 5c4d234392284a09b09d116964d4bc29 P a1ae730e8121439e940d5a61d67ddeff: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260925 07:29:57.967841 11099 consensus_queue.cc:260] T 5c4d234392284a09b09d116964d4bc29 P 25ae7fefa24c400f8a5e752d3845fdea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "724be09c3ade4d0d9043cce4e15a7e26" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 40653 } } peers { permanent_uuid: "25ae7fefa24c400f8a5e752d3845fdea" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 34311 } } peers { permanent_uuid: "a1ae730e8121439e940d5a61d67ddeff" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 40999 } }
I20260925 07:29:57.967927 11099 ts_tablet_manager.cc:1428] T 5c4d234392284a09b09d116964d4bc29 P 25ae7fefa24c400f8a5e752d3845fdea: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 07:29:57.967936 11101 raft_consensus.cc:348] T 5c4d234392284a09b09d116964d4bc29 P a1ae730e8121439e940d5a61d67ddeff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "724be09c3ade4d0d9043cce4e15a7e26" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 40653 } } peers { permanent_uuid: "25ae7fefa24c400f8a5e752d3845fdea" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 34311 } } peers { permanent_uuid: "a1ae730e8121439e940d5a61d67ddeff" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 40999 } }
I20260925 07:29:57.968020 11101 raft_consensus.cc:374] T 5c4d234392284a09b09d116964d4bc29 P a1ae730e8121439e940d5a61d67ddeff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:29:57.968042 11101 raft_consensus.cc:729] T 5c4d234392284a09b09d116964d4bc29 P a1ae730e8121439e940d5a61d67ddeff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1ae730e8121439e940d5a61d67ddeff, State: Initialized, Role: FOLLOWER
I20260925 07:29:57.968274 10579 catalog_manager.cc:4841] TS 724be09c3ade4d0d9043cce4e15a7e26 (127.9.66.1:40653): tablet f575e6c6766c4d71aaba804fdd265f1c (table test-table [id=389078d4503c4f0bb76b356a23629283]) successfully deleted
I20260925 07:29:57.968091 11101 consensus_queue.cc:260] T 5c4d234392284a09b09d116964d4bc29 P a1ae730e8121439e940d5a61d67ddeff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "724be09c3ade4d0d9043cce4e15a7e26" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 40653 } } peers { permanent_uuid: "25ae7fefa24c400f8a5e752d3845fdea" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 34311 } } peers { permanent_uuid: "a1ae730e8121439e940d5a61d67ddeff" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 40999 } }
I20260925 07:29:57.968412 11101 ts_tablet_manager.cc:1428] T 5c4d234392284a09b09d116964d4bc29 P a1ae730e8121439e940d5a61d67ddeff: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 07:29:57.971539 11106 raft_consensus.cc:482] T 5c4d234392284a09b09d116964d4bc29 P a1ae730e8121439e940d5a61d67ddeff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 07:29:57.971602 11106 raft_consensus.cc:504] T 5c4d234392284a09b09d116964d4bc29 P a1ae730e8121439e940d5a61d67ddeff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "724be09c3ade4d0d9043cce4e15a7e26" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 40653 } } peers { permanent_uuid: "25ae7fefa24c400f8a5e752d3845fdea" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 34311 } } peers { permanent_uuid: "a1ae730e8121439e940d5a61d67ddeff" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 40999 } }
I20260925 07:29:57.971923 11106 leader_election.cc:288] T 5c4d234392284a09b09d116964d4bc29 P a1ae730e8121439e940d5a61d67ddeff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 724be09c3ade4d0d9043cce4e15a7e26 (127.9.66.1:40653), 25ae7fefa24c400f8a5e752d3845fdea (127.9.66.2:34311)
I20260925 07:29:57.975502 11096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:29:57.972139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.3:56781 (local address 127.9.66.1:40653)
0925 07:29:57.972177 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 07:29:57.972179 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:29:57.972189 (+ 10us) server_negotiation.cc:407] Connection header received
0925 07:29:57.972213 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:29:57.972214 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:29:57.972222 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:29:57.972277 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:29:57.972420 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:29:57.973145 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:29:57.974627 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:29:57.975164 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:29:57.975225 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:29:57.975234 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 07:29:57.975407 (+ 173us) server_negotiation.cc:299] Negotiation successful
0925 07:29:57.975443 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 07:29:57.975507 10968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:29:57.972042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.1:40653 (local address 127.9.66.3:56781)
0925 07:29:57.972097 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0925 07:29:57.972109 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 07:29:57.972153 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:29:57.972283 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:29:57.972284 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:29:57.972285 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:29:57.972346 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:29:57.972348 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:29:57.973233 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:29:57.973256 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:29:57.974535 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:29:57.974538 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:29:57.975243 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:29:57.975244 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:29:57.975327 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:29:57.975331 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 07:29:57.975384 (+ 53us) client_negotiation.cc:240] Negotiation successful
0925 07:29:57.975421 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":64}
I20260925 07:29:57.975602 11095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:29:57.972171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.3:33909 (local address 127.9.66.2:34311)
0925 07:29:57.972206 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0925 07:29:57.972208 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:29:57.972402 (+ 194us) server_negotiation.cc:407] Connection header received
0925 07:29:57.972462 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:29:57.972463 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:29:57.972471 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:29:57.972522 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:29:57.972679 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:29:57.973112 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:29:57.974676 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:29:57.975233 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:29:57.975273 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:29:57.975280 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0925 07:29:57.975497 (+ 217us) server_negotiation.cc:299] Negotiation successful
0925 07:29:57.975535 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 07:29:57.975731 10749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c4d234392284a09b09d116964d4bc29" candidate_uuid: "a1ae730e8121439e940d5a61d67ddeff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "724be09c3ade4d0d9043cce4e15a7e26" is_pre_election: true
I20260925 07:29:57.975773 11126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:29:57.972060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.2:34311 (local address 127.9.66.3:33909)
0925 07:29:57.972330 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0925 07:29:57.972339 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 07:29:57.972414 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:29:57.972525 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:29:57.972526 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:29:57.972527 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:29:57.972599 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:29:57.972601 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:29:57.973200 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:29:57.973201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:29:57.974567 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:29:57.974570 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:29:57.975348 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:29:57.975349 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:29:57.975421 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:29:57.975423 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 07:29:57.975661 (+ 238us) client_negotiation.cc:240] Negotiation successful
0925 07:29:57.975698 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"mutex_wait_us":92,"thread_start_us":37,"threads_started":1}
I20260925 07:29:57.975806 10749 raft_consensus.cc:2454] T 5c4d234392284a09b09d116964d4bc29 P 724be09c3ade4d0d9043cce4e15a7e26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1ae730e8121439e940d5a61d67ddeff in term 0.
I20260925 07:29:57.975970 10963 leader_election.cc:302] T 5c4d234392284a09b09d116964d4bc29 P a1ae730e8121439e940d5a61d67ddeff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 724be09c3ade4d0d9043cce4e15a7e26, a1ae730e8121439e940d5a61d67ddeff; no voters:
I20260925 07:29:57.976071 11106 raft_consensus.cc:2781] T 5c4d234392284a09b09d116964d4bc29 P a1ae730e8121439e940d5a61d67ddeff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 07:29:57.976109 11106 raft_consensus.cc:482] T 5c4d234392284a09b09d116964d4bc29 P a1ae730e8121439e940d5a61d67ddeff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 07:29:57.976068 10889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c4d234392284a09b09d116964d4bc29" candidate_uuid: "a1ae730e8121439e940d5a61d67ddeff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25ae7fefa24c400f8a5e752d3845fdea" is_pre_election: true
I20260925 07:29:57.976135 11106 raft_consensus.cc:3037] T 5c4d234392284a09b09d116964d4bc29 P a1ae730e8121439e940d5a61d67ddeff [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:29:57.976157 10889 raft_consensus.cc:2454] T 5c4d234392284a09b09d116964d4bc29 P 25ae7fefa24c400f8a5e752d3845fdea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1ae730e8121439e940d5a61d67ddeff in term 0.
I20260925 07:29:57.976893 11106 raft_consensus.cc:504] T 5c4d234392284a09b09d116964d4bc29 P a1ae730e8121439e940d5a61d67ddeff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "724be09c3ade4d0d9043cce4e15a7e26" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 40653 } } peers { permanent_uuid: "25ae7fefa24c400f8a5e752d3845fdea" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 34311 } } peers { permanent_uuid: "a1ae730e8121439e940d5a61d67ddeff" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 40999 } }
I20260925 07:29:57.977021 11106 leader_election.cc:288] T 5c4d234392284a09b09d116964d4bc29 P a1ae730e8121439e940d5a61d67ddeff [CANDIDATE]: Term 1 election: Requested vote from peers 724be09c3ade4d0d9043cce4e15a7e26 (127.9.66.1:40653), 25ae7fefa24c400f8a5e752d3845fdea (127.9.66.2:34311)
I20260925 07:29:57.977131 10749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c4d234392284a09b09d116964d4bc29" candidate_uuid: "a1ae730e8121439e940d5a61d67ddeff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "724be09c3ade4d0d9043cce4e15a7e26"
I20260925 07:29:57.977190 10749 raft_consensus.cc:3037] T 5c4d234392284a09b09d116964d4bc29 P 724be09c3ade4d0d9043cce4e15a7e26 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:29:57.977195 10889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c4d234392284a09b09d116964d4bc29" candidate_uuid: "a1ae730e8121439e940d5a61d67ddeff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25ae7fefa24c400f8a5e752d3845fdea"
I20260925 07:29:57.977254 10889 raft_consensus.cc:3037] T 5c4d234392284a09b09d116964d4bc29 P 25ae7fefa24c400f8a5e752d3845fdea [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:29:57.977710 10749 raft_consensus.cc:2454] T 5c4d234392284a09b09d116964d4bc29 P 724be09c3ade4d0d9043cce4e15a7e26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1ae730e8121439e940d5a61d67ddeff in term 1.
I20260925 07:29:57.977845 10963 leader_election.cc:302] T 5c4d234392284a09b09d116964d4bc29 P a1ae730e8121439e940d5a61d67ddeff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 724be09c3ade4d0d9043cce4e15a7e26, a1ae730e8121439e940d5a61d67ddeff; no voters:
I20260925 07:29:57.977954 11106 raft_consensus.cc:2781] T 5c4d234392284a09b09d116964d4bc29 P a1ae730e8121439e940d5a61d67ddeff [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:29:57.978005 10889 raft_consensus.cc:2454] T 5c4d234392284a09b09d116964d4bc29 P 25ae7fefa24c400f8a5e752d3845fdea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1ae730e8121439e940d5a61d67ddeff in term 1.
I20260925 07:29:57.978083 11106 raft_consensus.cc:686] T 5c4d234392284a09b09d116964d4bc29 P a1ae730e8121439e940d5a61d67ddeff [term 1 LEADER]: Becoming Leader. State: Replica: a1ae730e8121439e940d5a61d67ddeff, State: Running, Role: LEADER
I20260925 07:29:57.978139 11106 consensus_queue.cc:237] T 5c4d234392284a09b09d116964d4bc29 P a1ae730e8121439e940d5a61d67ddeff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "724be09c3ade4d0d9043cce4e15a7e26" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 40653 } } peers { permanent_uuid: "25ae7fefa24c400f8a5e752d3845fdea" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 34311 } } peers { permanent_uuid: "a1ae730e8121439e940d5a61d67ddeff" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 40999 } }
I20260925 07:29:57.978768 10591 catalog_manager.cc:5495] T 5c4d234392284a09b09d116964d4bc29 P a1ae730e8121439e940d5a61d67ddeff reported cstate change: term changed from 0 to 1, leader changed from <none> to a1ae730e8121439e940d5a61d67ddeff (127.9.66.3). New cstate: current_term: 1 leader_uuid: "a1ae730e8121439e940d5a61d67ddeff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "724be09c3ade4d0d9043cce4e15a7e26" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 40653 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "25ae7fefa24c400f8a5e752d3845fdea" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 34311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1ae730e8121439e940d5a61d67ddeff" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 40999 } health_report { overall_health: HEALTHY } } }
Sep 25 07:29:57 dist-test-slave-57fn krb5kdc[10547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321397, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.62@KRBTEST.COM for kudu/127.9.66.62@KRBTEST.COM
I20260925 07:29:58.006281 11134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:29:57.998215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:37099 (local address 127.0.0.1:46156)
0925 07:29:57.998397 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0925 07:29:57.998410 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 07:29:57.998480 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:29:57.998637 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:29:57.998638 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:29:57.999034 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:29:57.999086 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:29:57.999089 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:29:57.999758 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:29:57.999759 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:29:58.000607 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:29:58.000610 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:29:58.000959 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:29:58.000960 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:29:58.001022 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:29:58.001731 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:29:58.001736 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:29:58.003344 (+ 1608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:29:58.005249 (+ 1905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:29:58.005254 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:29:58.005256 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:29:58.005452 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:29:58.005680 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:29:58.005680 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:29:58.005681 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:29:58.005744 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:29:58.006010 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:29:58.006015 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:29:58.006056 (+ 41us) client_negotiation.cc:769] Sending connection context
0925 07:29:58.006114 (+ 58us) client_negotiation.cc:240] Negotiation successful
0925 07:29:58.006207 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":105,"threads_started":1}
I20260925 07:29:58.006567 10642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:29:57.998264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46156 (local address 127.9.66.62:37099)
0925 07:29:57.998340 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0925 07:29:57.998342 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:29:57.998477 (+ 135us) server_negotiation.cc:407] Connection header received
0925 07:29:57.998546 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:29:57.998546 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:29:57.998555 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:29:57.998615 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:29:57.999167 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:29:57.999673 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:29:58.000691 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:29:58.000878 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:29:58.000935 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:29:58.003431 (+ 2496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:29:58.003432 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:29:58.003433 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:29:58.003440 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:29:58.005165 (+ 1725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:29:58.005552 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:29:58.005554 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:29:58.005554 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:29:58.005614 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:29:58.005807 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:29:58.005808 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:29:58.005808 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:29:58.005946 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:29:58.006026 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 07:29:58.006158 (+ 132us) server_negotiation.cc:299] Negotiation successful
0925 07:29:58.006267 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":32,"server-negotiator.queue_time_us":51}
I20260925 07:29:58.008536 10591 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.9.66.62@KRBTEST.COM'} at 127.0.0.1:46156:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 07:29:58.008749 10591 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 07:29:58.010164 10869 tablet_service.cc:1467] Processing CreateTablet for tablet 71a9e2ddee9a457bb95ef3f3c08becd5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8d2efca6020846a0a4ef093ddc33cd17]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 07:29:58.010254 10729 tablet_service.cc:1467] Processing CreateTablet for tablet 71a9e2ddee9a457bb95ef3f3c08becd5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8d2efca6020846a0a4ef093ddc33cd17]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 07:29:58.010336 10869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71a9e2ddee9a457bb95ef3f3c08becd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:29:58.010306 11009 tablet_service.cc:1467] Processing CreateTablet for tablet 71a9e2ddee9a457bb95ef3f3c08becd5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8d2efca6020846a0a4ef093ddc33cd17]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 07:29:58.010389 10729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71a9e2ddee9a457bb95ef3f3c08becd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:29:58.010437 11009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71a9e2ddee9a457bb95ef3f3c08becd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:29:58.011817 11099 tablet_bootstrap.cc:492] T 71a9e2ddee9a457bb95ef3f3c08becd5 P 25ae7fefa24c400f8a5e752d3845fdea: Bootstrap starting.
I20260925 07:29:58.012248 11099 tablet_bootstrap.cc:654] T 71a9e2ddee9a457bb95ef3f3c08becd5 P 25ae7fefa24c400f8a5e752d3845fdea: Neither blocks nor log segments found. Creating new log.
I20260925 07:29:58.012566 11098 tablet_bootstrap.cc:492] T 71a9e2ddee9a457bb95ef3f3c08becd5 P 724be09c3ade4d0d9043cce4e15a7e26: Bootstrap starting.
I20260925 07:29:58.012971 11099 tablet_bootstrap.cc:492] T 71a9e2ddee9a457bb95ef3f3c08becd5 P 25ae7fefa24c400f8a5e752d3845fdea: No bootstrap required, opened a new log
I20260925 07:29:58.012972 11098 tablet_bootstrap.cc:654] T 71a9e2ddee9a457bb95ef3f3c08becd5 P 724be09c3ade4d0d9043cce4e15a7e26: Neither blocks nor log segments found. Creating new log.
I20260925 07:29:58.013041 11099 ts_tablet_manager.cc:1397] T 71a9e2ddee9a457bb95ef3f3c08becd5 P 25ae7fefa24c400f8a5e752d3845fdea: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 07:29:58.013192 11099 raft_consensus.cc:348] T 71a9e2ddee9a457bb95ef3f3c08becd5 P 25ae7fefa24c400f8a5e752d3845fdea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "724be09c3ade4d0d9043cce4e15a7e26" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 40653 } } peers { permanent_uuid: "25ae7fefa24c400f8a5e752d3845fdea" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 34311 } } peers { permanent_uuid: "a1ae730e8121439e940d5a61d67ddeff" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 40999 } }
I20260925 07:29:58.013274 11099 raft_consensus.cc:374] T 71a9e2ddee9a457bb95ef3f3c08becd5 P 25ae7fefa24c400f8a5e752d3845fdea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:29:58.013302 11099 raft_consensus.cc:729] T 71a9e2ddee9a457bb95ef3f3c08becd5 P 25ae7fefa24c400f8a5e752d3845fdea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25ae7fefa24c400f8a5e752d3845fdea, State: Initialized, Role: FOLLOWER
I20260925 07:29:58.013315 11101 tablet_bootstrap.cc:492] T 71a9e2ddee9a457bb95ef3f3c08becd5 P a1ae730e8121439e940d5a61d67ddeff: Bootstrap starting.
I20260925 07:29:58.013345 11099 consensus_queue.cc:260] T 71a9e2ddee9a457bb95ef3f3c08becd5 P 25ae7fefa24c400f8a5e752d3845fdea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "724be09c3ade4d0d9043cce4e15a7e26" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 40653 } } peers { permanent_uuid: "25ae7fefa24c400f8a5e752d3845fdea" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 34311 } } peers { permanent_uuid: "a1ae730e8121439e940d5a61d67ddeff" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 40999 } }
I20260925 07:29:58.013453 11099 ts_tablet_manager.cc:1428] T 71a9e2ddee9a457bb95ef3f3c08becd5 P 25ae7fefa24c400f8a5e752d3845fdea: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 07:29:58.013557 11102 tablet_replica.cc:439] T 71a9e2ddee9a457bb95ef3f3c08becd5 P 25ae7fefa24c400f8a5e752d3845fdea: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "724be09c3ade4d0d9043cce4e15a7e26" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 40653 } } peers { permanent_uuid: "25ae7fefa24c400f8a5e752d3845fdea" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 34311 } } peers { permanent_uuid: "a1ae730e8121439e940d5a61d67ddeff" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 40999 } } }
I20260925 07:29:58.013645 11102 tablet_replica.cc:442] T 71a9e2ddee9a457bb95ef3f3c08becd5 P 25ae7fefa24c400f8a5e752d3845fdea: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 07:29:58.013708 11101 tablet_bootstrap.cc:654] T 71a9e2ddee9a457bb95ef3f3c08becd5 P a1ae730e8121439e940d5a61d67ddeff: Neither blocks nor log segments found. Creating new log.
I20260925 07:29:58.014423 11101 tablet_bootstrap.cc:492] T 71a9e2ddee9a457bb95ef3f3c08becd5 P a1ae730e8121439e940d5a61d67ddeff: No bootstrap required, opened a new log
I20260925 07:29:58.014488 11101 ts_tablet_manager.cc:1397] T 71a9e2ddee9a457bb95ef3f3c08becd5 P a1ae730e8121439e940d5a61d67ddeff: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 07:29:58.014578 11098 tablet_bootstrap.cc:492] T 71a9e2ddee9a457bb95ef3f3c08becd5 P 724be09c3ade4d0d9043cce4e15a7e26: No bootstrap required, opened a new log
I20260925 07:29:58.014629 11098 ts_tablet_manager.cc:1397] T 71a9e2ddee9a457bb95ef3f3c08becd5 P 724be09c3ade4d0d9043cce4e15a7e26: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 07:29:58.014638 11101 raft_consensus.cc:348] T 71a9e2ddee9a457bb95ef3f3c08becd5 P a1ae730e8121439e940d5a61d67ddeff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "724be09c3ade4d0d9043cce4e15a7e26" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 40653 } } peers { permanent_uuid: "25ae7fefa24c400f8a5e752d3845fdea" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 34311 } } peers { permanent_uuid: "a1ae730e8121439e940d5a61d67ddeff" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 40999 } }
I20260925 07:29:58.014710 11101 raft_consensus.cc:374] T 71a9e2ddee9a457bb95ef3f3c08becd5 P a1ae730e8121439e940d5a61d67ddeff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:29:58.014735 11101 raft_consensus.cc:729] T 71a9e2ddee9a457bb95ef3f3c08becd5 P a1ae730e8121439e940d5a61d67ddeff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1ae730e8121439e940d5a61d67ddeff, State: Initialized, Role: FOLLOWER
I20260925 07:29:58.014758 11098 raft_consensus.cc:348] T 71a9e2ddee9a457bb95ef3f3c08becd5 P 724be09c3ade4d0d9043cce4e15a7e26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "724be09c3ade4d0d9043cce4e15a7e26" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 40653 } } peers { permanent_uuid: "25ae7fefa24c400f8a5e752d3845fdea" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 34311 } } peers { permanent_uuid: "a1ae730e8121439e940d5a61d67ddeff" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 40999 } }
I20260925 07:29:58.014783 11101 consensus_queue.cc:260] T 71a9e2ddee9a457bb95ef3f3c08becd5 P a1ae730e8121439e940d5a61d67ddeff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "724be09c3ade4d0d9043cce4e15a7e26" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 40653 } } peers { permanent_uuid: "25ae7fefa24c400f8a5e752d3845fdea" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 34311 } } peers { permanent_uuid: "a1ae730e8121439e940d5a61d67ddeff" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 40999 } }
I20260925 07:29:58.014823 11098 raft_consensus.cc:374] T 71a9e2ddee9a457bb95ef3f3c08becd5 P 724be09c3ade4d0d9043cce4e15a7e26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:29:58.014849 11098 raft_consensus.cc:729] T 71a9e2ddee9a457bb95ef3f3c08becd5 P 724be09c3ade4d0d9043cce4e15a7e26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 724be09c3ade4d0d9043cce4e15a7e26, State: Initialized, Role: FOLLOWER
I20260925 07:29:58.014861 11101 ts_tablet_manager.cc:1428] T 71a9e2ddee9a457bb95ef3f3c08becd5 P a1ae730e8121439e940d5a61d67ddeff: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 07:29:58.014899 11098 consensus_queue.cc:260] T 71a9e2ddee9a457bb95ef3f3c08becd5 P 724be09c3ade4d0d9043cce4e15a7e26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "724be09c3ade4d0d9043cce4e15a7e26" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 40653 } } peers { permanent_uuid: "25ae7fefa24c400f8a5e752d3845fdea" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 34311 } } peers { permanent_uuid: "a1ae730e8121439e940d5a61d67ddeff" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 40999 } }
I20260925 07:29:58.014981 11098 ts_tablet_manager.cc:1428] T 71a9e2ddee9a457bb95ef3f3c08becd5 P 724be09c3ade4d0d9043cce4e15a7e26: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 07:29:58.014982 11106 tablet_replica.cc:439] T 71a9e2ddee9a457bb95ef3f3c08becd5 P a1ae730e8121439e940d5a61d67ddeff: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "724be09c3ade4d0d9043cce4e15a7e26" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 40653 } } peers { permanent_uuid: "25ae7fefa24c400f8a5e752d3845fdea" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 34311 } } peers { permanent_uuid: "a1ae730e8121439e940d5a61d67ddeff" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 40999 } } }
I20260925 07:29:58.015057 11106 tablet_replica.cc:442] T 71a9e2ddee9a457bb95ef3f3c08becd5 P a1ae730e8121439e940d5a61d67ddeff: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 07:29:58.015120 11105 tablet_replica.cc:439] T 71a9e2ddee9a457bb95ef3f3c08becd5 P 724be09c3ade4d0d9043cce4e15a7e26: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "724be09c3ade4d0d9043cce4e15a7e26" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 40653 } } peers { permanent_uuid: "25ae7fefa24c400f8a5e752d3845fdea" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 34311 } } peers { permanent_uuid: "a1ae730e8121439e940d5a61d67ddeff" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 40999 } } }
I20260925 07:29:58.015192 11105 tablet_replica.cc:442] T 71a9e2ddee9a457bb95ef3f3c08becd5 P 724be09c3ade4d0d9043cce4e15a7e26: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 07:29:58.028340 11105 raft_consensus.cc:482] T 71a9e2ddee9a457bb95ef3f3c08becd5 P 724be09c3ade4d0d9043cce4e15a7e26 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 07:29:58.028421 11105 raft_consensus.cc:504] T 71a9e2ddee9a457bb95ef3f3c08becd5 P 724be09c3ade4d0d9043cce4e15a7e26 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "724be09c3ade4d0d9043cce4e15a7e26" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 40653 } } peers { permanent_uuid: "25ae7fefa24c400f8a5e752d3845fdea" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 34311 } } peers { permanent_uuid: "a1ae730e8121439e940d5a61d67ddeff" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 40999 } }
I20260925 07:29:58.028705 11105 leader_election.cc:288] T 71a9e2ddee9a457bb95ef3f3c08becd5 P 724be09c3ade4d0d9043cce4e15a7e26 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 25ae7fefa24c400f8a5e752d3845fdea (127.9.66.2:34311), a1ae730e8121439e940d5a61d67ddeff (127.9.66.3:40999)
I20260925 07:29:58.032298 11138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:29:58.028914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.3:40999 (local address 127.9.66.1:34853)
0925 07:29:58.029052 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 07:29:58.029062 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 07:29:58.029141 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:29:58.029269 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:29:58.029270 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:29:58.029271 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:29:58.029354 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:29:58.029355 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:29:58.030056 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:29:58.030058 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:29:58.031304 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:29:58.031307 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:29:58.032049 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:29:58.032050 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:29:58.032117 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:29:58.032121 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 07:29:58.032180 (+ 59us) client_negotiation.cc:240] Negotiation successful
0925 07:29:58.032217 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"mutex_wait_us":17,"thread_start_us":59,"threads_started":1}
I20260925 07:29:58.032307 11097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:29:58.028977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.1:34853 (local address 127.9.66.3:40999)
0925 07:29:58.029020 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 07:29:58.029022 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:29:58.029114 (+ 92us) server_negotiation.cc:407] Connection header received
0925 07:29:58.029204 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:29:58.029205 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:29:58.029214 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:29:58.029267 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:29:58.029433 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:29:58.029973 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:29:58.031422 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:29:58.031958 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:29:58.032036 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:29:58.032047 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 07:29:58.032201 (+ 154us) server_negotiation.cc:299] Negotiation successful
0925 07:29:58.032248 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 07:29:58.032447 11095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:29:58.029070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.1:58043 (local address 127.9.66.2:34311)
0925 07:29:58.029110 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 07:29:58.029112 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:29:58.029119 (+ 7us) server_negotiation.cc:407] Connection header received
0925 07:29:58.029142 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:29:58.029143 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:29:58.029154 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:29:58.029205 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:29:58.029355 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:29:58.030140 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:29:58.031514 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:29:58.032026 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:29:58.032081 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:29:58.032088 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0925 07:29:58.032327 (+ 239us) server_negotiation.cc:299] Negotiation successful
0925 07:29:58.032379 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 07:29:58.032656 11029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71a9e2ddee9a457bb95ef3f3c08becd5" candidate_uuid: "724be09c3ade4d0d9043cce4e15a7e26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1ae730e8121439e940d5a61d67ddeff" is_pre_election: true
I20260925 07:29:58.032696 10688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:29:58.028902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.2:34311 (local address 127.9.66.1:58043)
0925 07:29:58.028989 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0925 07:29:58.029001 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 07:29:58.029043 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:29:58.029228 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:29:58.029229 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:29:58.029231 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:29:58.029292 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:29:58.029294 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:29:58.030258 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:29:58.030260 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:29:58.031445 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:29:58.031447 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:29:58.032117 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:29:58.032120 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:29:58.032210 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:29:58.032214 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 07:29:58.032259 (+ 45us) client_negotiation.cc:240] Negotiation successful
0925 07:29:58.032296 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260925 07:29:58.032738 11029 raft_consensus.cc:2454] T 71a9e2ddee9a457bb95ef3f3c08becd5 P a1ae730e8121439e940d5a61d67ddeff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 724be09c3ade4d0d9043cce4e15a7e26 in term 0.
I20260925 07:29:58.032893 10681 leader_election.cc:302] T 71a9e2ddee9a457bb95ef3f3c08becd5 P 724be09c3ade4d0d9043cce4e15a7e26 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 724be09c3ade4d0d9043cce4e15a7e26, a1ae730e8121439e940d5a61d67ddeff; no voters:
I20260925 07:29:58.032995 11105 raft_consensus.cc:2781] T 71a9e2ddee9a457bb95ef3f3c08becd5 P 724be09c3ade4d0d9043cce4e15a7e26 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 07:29:58.033031 11105 raft_consensus.cc:482] T 71a9e2ddee9a457bb95ef3f3c08becd5 P 724be09c3ade4d0d9043cce4e15a7e26 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 07:29:58.033002 10889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71a9e2ddee9a457bb95ef3f3c08becd5" candidate_uuid: "724be09c3ade4d0d9043cce4e15a7e26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25ae7fefa24c400f8a5e752d3845fdea" is_pre_election: true
I20260925 07:29:58.033059 11105 raft_consensus.cc:3037] T 71a9e2ddee9a457bb95ef3f3c08becd5 P 724be09c3ade4d0d9043cce4e15a7e26 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:29:58.033072 10889 raft_consensus.cc:2454] T 71a9e2ddee9a457bb95ef3f3c08becd5 P 25ae7fefa24c400f8a5e752d3845fdea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 724be09c3ade4d0d9043cce4e15a7e26 in term 0.
I20260925 07:29:58.033762 11105 raft_consensus.cc:504] T 71a9e2ddee9a457bb95ef3f3c08becd5 P 724be09c3ade4d0d9043cce4e15a7e26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "724be09c3ade4d0d9043cce4e15a7e26" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 40653 } } peers { permanent_uuid: "25ae7fefa24c400f8a5e752d3845fdea" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 34311 } } peers { permanent_uuid: "a1ae730e8121439e940d5a61d67ddeff" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 40999 } }
I20260925 07:29:58.033916 11105 leader_election.cc:288] T 71a9e2ddee9a457bb95ef3f3c08becd5 P 724be09c3ade4d0d9043cce4e15a7e26 [CANDIDATE]: Term 1 election: Requested vote from peers 25ae7fefa24c400f8a5e752d3845fdea (127.9.66.2:34311), a1ae730e8121439e940d5a61d67ddeff (127.9.66.3:40999)
I20260925 07:29:58.034057 11029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71a9e2ddee9a457bb95ef3f3c08becd5" candidate_uuid: "724be09c3ade4d0d9043cce4e15a7e26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1ae730e8121439e940d5a61d67ddeff"
I20260925 07:29:58.034132 11029 raft_consensus.cc:3037] T 71a9e2ddee9a457bb95ef3f3c08becd5 P a1ae730e8121439e940d5a61d67ddeff [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:29:58.034106 10889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71a9e2ddee9a457bb95ef3f3c08becd5" candidate_uuid: "724be09c3ade4d0d9043cce4e15a7e26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25ae7fefa24c400f8a5e752d3845fdea"
I20260925 07:29:58.034169 10889 raft_consensus.cc:3037] T 71a9e2ddee9a457bb95ef3f3c08becd5 P 25ae7fefa24c400f8a5e752d3845fdea [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:29:58.034710 10889 raft_consensus.cc:2454] T 71a9e2ddee9a457bb95ef3f3c08becd5 P 25ae7fefa24c400f8a5e752d3845fdea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 724be09c3ade4d0d9043cce4e15a7e26 in term 1.
I20260925 07:29:58.034875 10680 leader_election.cc:302] T 71a9e2ddee9a457bb95ef3f3c08becd5 P 724be09c3ade4d0d9043cce4e15a7e26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25ae7fefa24c400f8a5e752d3845fdea, 724be09c3ade4d0d9043cce4e15a7e26; no voters:
I20260925 07:29:58.034977 11105 raft_consensus.cc:2781] T 71a9e2ddee9a457bb95ef3f3c08becd5 P 724be09c3ade4d0d9043cce4e15a7e26 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:29:58.035118 11105 raft_consensus.cc:686] T 71a9e2ddee9a457bb95ef3f3c08becd5 P 724be09c3ade4d0d9043cce4e15a7e26 [term 1 LEADER]: Becoming Leader. State: Replica: 724be09c3ade4d0d9043cce4e15a7e26, State: Running, Role: LEADER
I20260925 07:29:58.035151 11029 raft_consensus.cc:2454] T 71a9e2ddee9a457bb95ef3f3c08becd5 P a1ae730e8121439e940d5a61d67ddeff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 724be09c3ade4d0d9043cce4e15a7e26 in term 1.
I20260925 07:29:58.035187 11105 consensus_queue.cc:237] T 71a9e2ddee9a457bb95ef3f3c08becd5 P 724be09c3ade4d0d9043cce4e15a7e26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "724be09c3ade4d0d9043cce4e15a7e26" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 40653 } } peers { permanent_uuid: "25ae7fefa24c400f8a5e752d3845fdea" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 34311 } } peers { permanent_uuid: "a1ae730e8121439e940d5a61d67ddeff" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 40999 } }
I20260925 07:29:58.035503 11139 tablet_replica.cc:439] T 71a9e2ddee9a457bb95ef3f3c08becd5 P 724be09c3ade4d0d9043cce4e15a7e26: TxnStatusTablet state changed. Reason: New leader 724be09c3ade4d0d9043cce4e15a7e26. Latest consensus state: current_term: 1 leader_uuid: "724be09c3ade4d0d9043cce4e15a7e26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "724be09c3ade4d0d9043cce4e15a7e26" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 40653 } } peers { permanent_uuid: "25ae7fefa24c400f8a5e752d3845fdea" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 34311 } } peers { permanent_uuid: "a1ae730e8121439e940d5a61d67ddeff" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 40999 } } }
I20260925 07:29:58.035658 11139 tablet_replica.cc:442] T 71a9e2ddee9a457bb95ef3f3c08becd5 P 724be09c3ade4d0d9043cce4e15a7e26: This TxnStatusTablet replica's current role is: LEADER
I20260925 07:29:58.035777 10592 catalog_manager.cc:5495] T 71a9e2ddee9a457bb95ef3f3c08becd5 P 724be09c3ade4d0d9043cce4e15a7e26 reported cstate change: term changed from 0 to 1, leader changed from <none> to 724be09c3ade4d0d9043cce4e15a7e26 (127.9.66.1). New cstate: current_term: 1 leader_uuid: "724be09c3ade4d0d9043cce4e15a7e26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "724be09c3ade4d0d9043cce4e15a7e26" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 40653 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "25ae7fefa24c400f8a5e752d3845fdea" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 34311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1ae730e8121439e940d5a61d67ddeff" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 40999 } health_report { overall_health: UNKNOWN } } }
I20260925 07:29:58.035847 11141 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 07:29:58.035967 11141 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 07:29:58.047333 11134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:29:58.044075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.1:40653 (local address 127.0.0.1:48980)
0925 07:29:58.044188 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0925 07:29:58.044201 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 07:29:58.044242 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:29:58.044414 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:29:58.044414 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:29:58.044415 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 07:29:58.044457 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:29:58.044459 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:29:58.045379 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:29:58.045380 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:29:58.046313 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:29:58.046315 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:29:58.046801 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:29:58.046802 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:29:58.046863 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:29:58.046878 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 07:29:58.047165 (+ 287us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 07:29:58.047171 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 07:29:58.047210 (+ 39us) client_negotiation.cc:240] Negotiation successful
0925 07:29:58.047256 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79}
I20260925 07:29:58.047766 11096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:29:58.044205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48980 (local address 127.9.66.1:40653)
0925 07:29:58.044247 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 07:29:58.044249 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:29:58.044258 (+ 9us) server_negotiation.cc:407] Connection header received
0925 07:29:58.044327 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:29:58.044328 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:29:58.044344 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:29:58.044410 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 07:29:58.044539 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:29:58.045284 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:29:58.046419 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:29:58.046710 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:29:58.046766 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:29:58.046976 (+ 210us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 07:29:58.047083 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 07:29:58.047142 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 07:29:58.047684 (+ 542us) server_negotiation.cc:299] Negotiation successful
0925 07:29:58.047702 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 07:29:58.048678 11029 raft_consensus.cc:1261] T 71a9e2ddee9a457bb95ef3f3c08becd5 P a1ae730e8121439e940d5a61d67ddeff [term 1 FOLLOWER]: Refusing update from remote peer 724be09c3ade4d0d9043cce4e15a7e26: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:29:58.048714 10889 raft_consensus.cc:1261] T 71a9e2ddee9a457bb95ef3f3c08becd5 P 25ae7fefa24c400f8a5e752d3845fdea [term 1 FOLLOWER]: Refusing update from remote peer 724be09c3ade4d0d9043cce4e15a7e26: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:29:58.048914 11105 consensus_queue.cc:1035] T 71a9e2ddee9a457bb95ef3f3c08becd5 P 724be09c3ade4d0d9043cce4e15a7e26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1ae730e8121439e940d5a61d67ddeff" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 40999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:29:58.049002 11139 consensus_queue.cc:1035] T 71a9e2ddee9a457bb95ef3f3c08becd5 P 724be09c3ade4d0d9043cce4e15a7e26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25ae7fefa24c400f8a5e752d3845fdea" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 34311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:29:58.049605 11102 tablet_replica.cc:439] T 71a9e2ddee9a457bb95ef3f3c08becd5 P 25ae7fefa24c400f8a5e752d3845fdea: TxnStatusTablet state changed. Reason: New leader 724be09c3ade4d0d9043cce4e15a7e26. Latest consensus state: current_term: 1 leader_uuid: "724be09c3ade4d0d9043cce4e15a7e26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "724be09c3ade4d0d9043cce4e15a7e26" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 40653 } } peers { permanent_uuid: "25ae7fefa24c400f8a5e752d3845fdea" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 34311 } } peers { permanent_uuid: "a1ae730e8121439e940d5a61d67ddeff" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 40999 } } }
I20260925 07:29:58.049712 11102 tablet_replica.cc:442] T 71a9e2ddee9a457bb95ef3f3c08becd5 P 25ae7fefa24c400f8a5e752d3845fdea: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 07:29:58.049818 11106 tablet_replica.cc:439] T 71a9e2ddee9a457bb95ef3f3c08becd5 P a1ae730e8121439e940d5a61d67ddeff: TxnStatusTablet state changed. Reason: New leader 724be09c3ade4d0d9043cce4e15a7e26. Latest consensus state: current_term: 1 leader_uuid: "724be09c3ade4d0d9043cce4e15a7e26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "724be09c3ade4d0d9043cce4e15a7e26" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 40653 } } peers { permanent_uuid: "25ae7fefa24c400f8a5e752d3845fdea" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 34311 } } peers { permanent_uuid: "a1ae730e8121439e940d5a61d67ddeff" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 40999 } } }
I20260925 07:29:58.049894 11106 tablet_replica.cc:442] T 71a9e2ddee9a457bb95ef3f3c08becd5 P a1ae730e8121439e940d5a61d67ddeff: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 07:29:58.054189 11139 tablet_replica.cc:439] T 71a9e2ddee9a457bb95ef3f3c08becd5 P 724be09c3ade4d0d9043cce4e15a7e26: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "724be09c3ade4d0d9043cce4e15a7e26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "724be09c3ade4d0d9043cce4e15a7e26" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 40653 } } peers { permanent_uuid: "25ae7fefa24c400f8a5e752d3845fdea" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 34311 } } peers { permanent_uuid: "a1ae730e8121439e940d5a61d67ddeff" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 40999 } } }
I20260925 07:29:58.054270 11139 tablet_replica.cc:442] T 71a9e2ddee9a457bb95ef3f3c08becd5 P 724be09c3ade4d0d9043cce4e15a7e26: This TxnStatusTablet replica's current role is: LEADER
I20260925 07:29:58.054358 11139 tablet_replica.cc:439] T 71a9e2ddee9a457bb95ef3f3c08becd5 P 724be09c3ade4d0d9043cce4e15a7e26: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "724be09c3ade4d0d9043cce4e15a7e26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "724be09c3ade4d0d9043cce4e15a7e26" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 40653 } } peers { permanent_uuid: "25ae7fefa24c400f8a5e752d3845fdea" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 34311 } } peers { permanent_uuid: "a1ae730e8121439e940d5a61d67ddeff" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 40999 } } }
I20260925 07:29:58.054401 11139 tablet_replica.cc:442] T 71a9e2ddee9a457bb95ef3f3c08becd5 P 724be09c3ade4d0d9043cce4e15a7e26: This TxnStatusTablet replica's current role is: LEADER
I20260925 07:29:58.054477 11112 mvcc.cc:205] Tried to move back new op lower bound from 7333156446405619712 to 7333156446352523264. Current Snapshot: MvccSnapshot[applied={T|T < 7333156446405619712 or (T in {7333156446405619712})}]
I20260925 07:29:58.054811 11102 tablet_replica.cc:439] T 71a9e2ddee9a457bb95ef3f3c08becd5 P 25ae7fefa24c400f8a5e752d3845fdea: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "724be09c3ade4d0d9043cce4e15a7e26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "724be09c3ade4d0d9043cce4e15a7e26" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 40653 } } peers { permanent_uuid: "25ae7fefa24c400f8a5e752d3845fdea" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 34311 } } peers { permanent_uuid: "a1ae730e8121439e940d5a61d67ddeff" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 40999 } } }
I20260925 07:29:58.054908 11102 tablet_replica.cc:442] T 71a9e2ddee9a457bb95ef3f3c08becd5 P 25ae7fefa24c400f8a5e752d3845fdea: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 07:29:58.055104 11106 tablet_replica.cc:439] T 71a9e2ddee9a457bb95ef3f3c08becd5 P a1ae730e8121439e940d5a61d67ddeff: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "724be09c3ade4d0d9043cce4e15a7e26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "724be09c3ade4d0d9043cce4e15a7e26" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 40653 } } peers { permanent_uuid: "25ae7fefa24c400f8a5e752d3845fdea" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 34311 } } peers { permanent_uuid: "a1ae730e8121439e940d5a61d67ddeff" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 40999 } } }
I20260925 07:29:58.055186 11106 tablet_replica.cc:442] T 71a9e2ddee9a457bb95ef3f3c08becd5 P a1ae730e8121439e940d5a61d67ddeff: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 07:29:58.059126 11097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:29:58.056240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53870 (local address 127.9.66.3:40999)
0925 07:29:58.056272 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0925 07:29:58.056274 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:29:58.056298 (+ 24us) server_negotiation.cc:407] Connection header received
0925 07:29:58.056368 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:29:58.056369 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:29:58.056392 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:29:58.056462 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 07:29:58.056596 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:29:58.057287 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:29:58.058302 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:29:58.058592 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:29:58.058657 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:29:58.058819 (+ 162us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 07:29:58.058924 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 07:29:58.058988 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 07:29:58.059048 (+ 60us) server_negotiation.cc:299] Negotiation successful
0925 07:29:58.059065 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 07:29:58.064666 11096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:29:58.061744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.3:46183 (local address 127.9.66.1:40653)
0925 07:29:58.061801 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 07:29:58.061803 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:29:58.061811 (+ 8us) server_negotiation.cc:407] Connection header received
0925 07:29:58.061831 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:29:58.061831 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:29:58.061840 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:29:58.061896 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 07:29:58.062079 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:29:58.062765 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:29:58.063872 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:29:58.064163 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:29:58.064231 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:29:58.064422 (+ 191us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 07:29:58.064456 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 07:29:58.064521 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 07:29:58.064586 (+ 65us) server_negotiation.cc:299] Negotiation successful
0925 07:29:58.064604 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 07:29:58.064702 11081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:29:58.061560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.1:40653 (local address 127.9.66.3:46183)
0925 07:29:58.061629 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0925 07:29:58.061639 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 07:29:58.061679 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:29:58.061922 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:29:58.061923 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:29:58.061924 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 07:29:58.061988 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:29:58.061989 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:29:58.062853 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:29:58.062854 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:29:58.063785 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:29:58.063788 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:29:58.064248 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:29:58.064249 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:29:58.064316 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:29:58.064329 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 07:29:58.064537 (+ 208us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 07:29:58.064539 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 07:29:58.064584 (+ 45us) client_negotiation.cc:240] Negotiation successful
0925 07:29:58.064626 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260925 07:29:58.067250 10889 raft_consensus.cc:1261] T 5c4d234392284a09b09d116964d4bc29 P 25ae7fefa24c400f8a5e752d3845fdea [term 1 FOLLOWER]: Refusing update from remote peer a1ae730e8121439e940d5a61d67ddeff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:29:58.067301 10749 raft_consensus.cc:1261] T 5c4d234392284a09b09d116964d4bc29 P 724be09c3ade4d0d9043cce4e15a7e26 [term 1 FOLLOWER]: Refusing update from remote peer a1ae730e8121439e940d5a61d67ddeff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:29:58.067468 11127 consensus_queue.cc:1035] T 5c4d234392284a09b09d116964d4bc29 P a1ae730e8121439e940d5a61d67ddeff [LEADER]: Connected to new peer: Peer: permanent_uuid: "25ae7fefa24c400f8a5e752d3845fdea" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 34311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:29:58.067613 11106 consensus_queue.cc:1035] T 5c4d234392284a09b09d116964d4bc29 P a1ae730e8121439e940d5a61d67ddeff [LEADER]: Connected to new peer: Peer: permanent_uuid: "724be09c3ade4d0d9043cce4e15a7e26" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 40653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 07:29:58.068553 10796 tablet.cc:2369] T 5c4d234392284a09b09d116964d4bc29 P 724be09c3ade4d0d9043cce4e15a7e26: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 07:29:58.080919 10802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:29:58.077904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.3:40999 (local address 127.9.66.1:44117)
0925 07:29:58.077963 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0925 07:29:58.077977 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 07:29:58.078021 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:29:58.078140 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:29:58.078141 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:29:58.078142 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 07:29:58.078200 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:29:58.078202 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:29:58.079123 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:29:58.079124 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:29:58.079999 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:29:58.080001 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:29:58.080444 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:29:58.080445 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:29:58.080530 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:29:58.080545 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 07:29:58.080748 (+ 203us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 07:29:58.080750 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 07:29:58.080804 (+ 54us) client_negotiation.cc:240] Negotiation successful
0925 07:29:58.080846 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22}
I20260925 07:29:58.080914 11097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:29:58.077992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.1:44117 (local address 127.9.66.3:40999)
0925 07:29:58.078027 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0925 07:29:58.078030 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:29:58.078040 (+ 10us) server_negotiation.cc:407] Connection header received
0925 07:29:58.078064 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:29:58.078064 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:29:58.078073 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:29:58.078128 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 07:29:58.078270 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:29:58.079033 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:29:58.080081 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:29:58.080358 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:29:58.080435 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:29:58.080637 (+ 202us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 07:29:58.080672 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 07:29:58.080729 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 07:29:58.080817 (+ 88us) server_negotiation.cc:299] Negotiation successful
0925 07:29:58.080839 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 07:29:58.090685 10591 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46152:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 07:29:58.090795 10591 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46152:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 07:29:58.091328 10591 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 42650a67401745e9be59eb4711a81af7: Sending DeleteTablet for 3 replicas of tablet 5c4d234392284a09b09d116964d4bc29
I20260925 07:29:58.091584 10729 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c4d234392284a09b09d116964d4bc29 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:29:58 UTC) from {username='kudu', principal='kudu/127.9.66.62@KRBTEST.COM'} at 127.0.0.1:48974
I20260925 07:29:58.091668 11122 tablet_replica.cc:330] T 5c4d234392284a09b09d116964d4bc29 P 724be09c3ade4d0d9043cce4e15a7e26: stopping tablet replica
I20260925 07:29:58.091715 11122 raft_consensus.cc:2229] T 5c4d234392284a09b09d116964d4bc29 P 724be09c3ade4d0d9043cce4e15a7e26 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 07:29:58.091758 11122 raft_consensus.cc:2258] T 5c4d234392284a09b09d116964d4bc29 P 724be09c3ade4d0d9043cce4e15a7e26 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:29:58.092042 11009 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c4d234392284a09b09d116964d4bc29 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:29:58 UTC) from {username='kudu', principal='kudu/127.9.66.62@KRBTEST.COM'} at 127.0.0.1:53860
I20260925 07:29:58.092121 11120 tablet_replica.cc:330] T 5c4d234392284a09b09d116964d4bc29 P a1ae730e8121439e940d5a61d67ddeff: stopping tablet replica
I20260925 07:29:58.092162 11120 raft_consensus.cc:2229] T 5c4d234392284a09b09d116964d4bc29 P a1ae730e8121439e940d5a61d67ddeff [term 1 LEADER]: Raft consensus shutting down.
I20260925 07:29:58.092233 11120 raft_consensus.cc:2258] T 5c4d234392284a09b09d116964d4bc29 P a1ae730e8121439e940d5a61d67ddeff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:29:58.093065 11120 ts_tablet_manager.cc:1905] T 5c4d234392284a09b09d116964d4bc29 P a1ae730e8121439e940d5a61d67ddeff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 07:29:58.094600 10869 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c4d234392284a09b09d116964d4bc29 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:29:58 UTC) from {username='kudu', principal='kudu/127.9.66.62@KRBTEST.COM'} at 127.0.0.1:56602
I20260925 07:29:58.094703 11121 tablet_replica.cc:330] T 5c4d234392284a09b09d116964d4bc29 P 25ae7fefa24c400f8a5e752d3845fdea: stopping tablet replica
I20260925 07:29:58.094738 11120 ts_tablet_manager.cc:1918] T 5c4d234392284a09b09d116964d4bc29 P a1ae730e8121439e940d5a61d67ddeff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 07:29:58.094781 11120 log.cc:1196] T 5c4d234392284a09b09d116964d4bc29 P a1ae730e8121439e940d5a61d67ddeff: Deleting WAL directory at /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790321392278483-9480-0/minicluster-data/ts-2/wal/wals/5c4d234392284a09b09d116964d4bc29
I20260925 07:29:58.094782 11121 raft_consensus.cc:2229] T 5c4d234392284a09b09d116964d4bc29 P 25ae7fefa24c400f8a5e752d3845fdea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 07:29:58.094820 11121 raft_consensus.cc:2258] T 5c4d234392284a09b09d116964d4bc29 P 25ae7fefa24c400f8a5e752d3845fdea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:29:58.095000 11120 ts_tablet_manager.cc:1939] T 5c4d234392284a09b09d116964d4bc29 P a1ae730e8121439e940d5a61d67ddeff: Deleting consensus metadata
I20260925 07:29:58.095127 11121 ts_tablet_manager.cc:1905] T 5c4d234392284a09b09d116964d4bc29 P 25ae7fefa24c400f8a5e752d3845fdea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 07:29:58.095266 10577 catalog_manager.cc:4841] TS a1ae730e8121439e940d5a61d67ddeff (127.9.66.3:40999): tablet 5c4d234392284a09b09d116964d4bc29 (table test-table [id=5eb3bfcabbeb4e91af451af20ac0ee63]) successfully deleted
I20260925 07:29:58.095505 11122 ts_tablet_manager.cc:1905] T 5c4d234392284a09b09d116964d4bc29 P 724be09c3ade4d0d9043cce4e15a7e26: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 07:29:58.096640 11121 ts_tablet_manager.cc:1918] T 5c4d234392284a09b09d116964d4bc29 P 25ae7fefa24c400f8a5e752d3845fdea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 07:29:58.096702 11121 log.cc:1196] T 5c4d234392284a09b09d116964d4bc29 P 25ae7fefa24c400f8a5e752d3845fdea: Deleting WAL directory at /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790321392278483-9480-0/minicluster-data/ts-1/wal/wals/5c4d234392284a09b09d116964d4bc29
I20260925 07:29:58.096886 11122 ts_tablet_manager.cc:1918] T 5c4d234392284a09b09d116964d4bc29 P 724be09c3ade4d0d9043cce4e15a7e26: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 07:29:58.096935 11121 ts_tablet_manager.cc:1939] T 5c4d234392284a09b09d116964d4bc29 P 25ae7fefa24c400f8a5e752d3845fdea: Deleting consensus metadata
I20260925 07:29:58.096948 11122 log.cc:1196] T 5c4d234392284a09b09d116964d4bc29 P 724be09c3ade4d0d9043cce4e15a7e26: Deleting WAL directory at /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790321392278483-9480-0/minicluster-data/ts-0/wal/wals/5c4d234392284a09b09d116964d4bc29
I20260925 07:29:58.097179 11122 ts_tablet_manager.cc:1939] T 5c4d234392284a09b09d116964d4bc29 P 724be09c3ade4d0d9043cce4e15a7e26: Deleting consensus metadata
I20260925 07:29:58.097240 10576 catalog_manager.cc:4841] TS 25ae7fefa24c400f8a5e752d3845fdea (127.9.66.2:34311): tablet 5c4d234392284a09b09d116964d4bc29 (table test-table [id=5eb3bfcabbeb4e91af451af20ac0ee63]) successfully deleted
I20260925 07:29:58.097482 10579 catalog_manager.cc:4841] TS 724be09c3ade4d0d9043cce4e15a7e26 (127.9.66.1:40653): tablet 5c4d234392284a09b09d116964d4bc29 (table test-table [id=5eb3bfcabbeb4e91af451af20ac0ee63]) successfully deleted
Sep 25 07:29:58 dist-test-slave-57fn krb5kdc[10547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321397, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.9.66.1@KRBTEST.COM
I20260925 07:29:58.107837 11096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:29:58.097795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48986 (local address 127.9.66.1:40653)
0925 07:29:58.097833 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 07:29:58.097835 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:29:58.099605 (+ 1770us) server_negotiation.cc:407] Connection header received
0925 07:29:58.099700 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:29:58.099701 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:29:58.099711 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:29:58.099739 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:29:58.100483 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:29:58.100945 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:29:58.101755 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:29:58.101939 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:29:58.102024 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:29:58.104902 (+ 2878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:29:58.104911 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:29:58.104912 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:29:58.104919 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:29:58.106589 (+ 1670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:29:58.106962 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:29:58.106965 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:29:58.106965 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:29:58.107030 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:29:58.107224 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:29:58.107225 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:29:58.107225 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:29:58.107361 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:29:58.107529 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0925 07:29:58.107570 (+ 41us) server_negotiation.cc:299] Negotiation successful
0925 07:29:58.107649 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
W20260925 07:29:58.108070 10789 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48986
I20260925 07:29:58.115144 10642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:29:58.108894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46160 (local address 127.9.66.62:37099)
0925 07:29:58.108939 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 07:29:58.108941 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:29:58.109095 (+ 154us) server_negotiation.cc:407] Connection header received
0925 07:29:58.109163 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:29:58.109164 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:29:58.109174 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:29:58.109230 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:29:58.109767 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:29:58.110260 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:29:58.110924 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:29:58.111125 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:29:58.111198 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:29:58.112306 (+ 1108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:29:58.112307 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:29:58.112308 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:29:58.112314 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:29:58.113963 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:29:58.114293 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:29:58.114295 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:29:58.114295 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:29:58.114347 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:29:58.114589 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:29:58.114590 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:29:58.114590 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:29:58.114699 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:29:58.114800 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 07:29:58.114962 (+ 162us) server_negotiation.cc:299] Negotiation successful
0925 07:29:58.115036 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
W20260925 07:29:58.115329 10591 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46160
I20260925 07:29:58.115960 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 10664
I20260925 07:29:58.121580 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 10804
I20260925 07:29:58.128409 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 10944
I20260925 07:29:58.133319 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 10560
2026-09-25T07:29:58Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2085 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790321392278483-9480-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 07:29:58 dist-test-slave-57fn krb5kdc[11161](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 07:29:58 dist-test-slave-57fn krb5kdc[11161](info): set up 2 sockets
Sep 25 07:29:58 dist-test-slave-57fn krb5kdc[11161](info): commencing operation
krb5kdc: starting...
W20260925 07:29:59.173292 9480 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790321392278483-9480-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790321392278483-9480-0/krb5kdc/test-user.keytab.
Sep 25 07:29:59 dist-test-slave-57fn krb5kdc[11161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321399, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T07:29:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T07:29:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.66.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.66.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.66.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab.
Entry for principal kudu/127.9.66.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab.
I20260925 07:29:59.284662 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790321392278483-9480-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790321392278483-9480-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790321392278483-9480-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.66.62:45659
--webserver_interface=127.9.66.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab
--principal=kudu/127.9.66.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:37843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.66.62:45659
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:29:59.361965 11174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:29:59.362200 11174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:29:59.362234 11174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:29:59.363353 11174 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:29:59.363426 11174 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 07:29:59.363451 11174 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:29:59.363484 11174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:29:59.363509 11174 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:29:59.363536 11174 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:29:59.364984 11174 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:37843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790321392278483-9480-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790321392278483-9480-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.66.62:45659
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab
--principal=kudu/127.9.66.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.66.62:45659
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.66.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11174
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790321392278483-9480-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:29:59.365247 11174 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:29:59.365459 11174 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:29:59.370168 11180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:29:59.370184 11179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:29:59.370198 11182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:29:59.370319 11174 server_base.cc:1034] running on GCE node
I20260925 07:29:59.370592 11174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:29:59.370884 11174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:29:59.372033 11174 hybrid_clock.cc:648] HybridClock initialized: now 1790321399372017 us; error 46 us; skew 500 ppm
Sep 25 07:29:59 dist-test-slave-57fn krb5kdc[11161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321399, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:29:59.379660 11174 init.cc:377] Logged in from keytab as kudu/127.9.66.62@KRBTEST.COM (short username kudu)
I20260925 07:29:59.380128 11174 webserver.cc:460] Webserver started at http://127.9.66.62:36383/ using document root <none> and password file <none>
I20260925 07:29:59.380360 11174 fs_manager.cc:359] Metadata directory not provided
I20260925 07:29:59.380406 11174 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:29:59.380553 11174 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:29:59.381659 11174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790321392278483-9480-0/minicluster-data/master-0/data/instance:
uuid: "c175dfccd77d4fcebebcdc0419cdd1d5"
format_stamp: "Formatted at 2026-09-25 07:29:59 on dist-test-slave-57fn"
server_key: "3205373907c9f6f4a97965fd3a027bb8"
server_key_iv: "77e8f24bfefa9091318ed1f217dead31"
server_key_version: "encryptionkey@0"
I20260925 07:29:59.382025 11174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790321392278483-9480-0/minicluster-data/master-0/wal/instance:
uuid: "c175dfccd77d4fcebebcdc0419cdd1d5"
format_stamp: "Formatted at 2026-09-25 07:29:59 on dist-test-slave-57fn"
server_key: "3205373907c9f6f4a97965fd3a027bb8"
server_key_iv: "77e8f24bfefa9091318ed1f217dead31"
server_key_version: "encryptionkey@0"
I20260925 07:29:59.383313 11174 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 07:29:59.384003 11189 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:29:59.384171 11174 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 07:29:59.384255 11174 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790321392278483-9480-0/minicluster-data/master-0/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790321392278483-9480-0/minicluster-data/master-0/wal
uuid: "c175dfccd77d4fcebebcdc0419cdd1d5"
format_stamp: "Formatted at 2026-09-25 07:29:59 on dist-test-slave-57fn"
server_key: "3205373907c9f6f4a97965fd3a027bb8"
server_key_iv: "77e8f24bfefa9091318ed1f217dead31"
server_key_version: "encryptionkey@0"
I20260925 07:29:59.384307 11174 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790321392278483-9480-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790321392278483-9480-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790321392278483-9480-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:29:59.397110 11174 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:29:59.399242 11174 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:29:59.399348 11174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:29:59.403579 11174 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.62:45659
I20260925 07:29:59.403692 11251 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.62:45659 every 8 connection(s)
I20260925 07:29:59.403997 11174 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
I20260925 07:29:59.404616 11252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:29:59.406764 11252 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c175dfccd77d4fcebebcdc0419cdd1d5: Bootstrap starting.
I20260925 07:29:59.407311 11252 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c175dfccd77d4fcebebcdc0419cdd1d5: Neither blocks nor log segments found. Creating new log.
I20260925 07:29:59.407608 11252 log.cc:824] T 00000000000000000000000000000000 P c175dfccd77d4fcebebcdc0419cdd1d5: Log is configured to *not* fsync() on all Append() calls
I20260925 07:29:59.408247 11252 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c175dfccd77d4fcebebcdc0419cdd1d5: No bootstrap required, opened a new log
I20260925 07:29:59.408762 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 11174
I20260925 07:29:59.408869 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790321392278483-9480-0/minicluster-data/master-0/wal/instance
I20260925 07:29:59.409031 9480 external_mini_cluster.cc:1428] Setting key 182f1d132de3dcde83534fd710285192
I20260925 07:29:59.410705 11252 raft_consensus.cc:348] T 00000000000000000000000000000000 P c175dfccd77d4fcebebcdc0419cdd1d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c175dfccd77d4fcebebcdc0419cdd1d5" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 45659 } }
I20260925 07:29:59.410859 11252 raft_consensus.cc:374] T 00000000000000000000000000000000 P c175dfccd77d4fcebebcdc0419cdd1d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:29:59.410920 11252 raft_consensus.cc:729] T 00000000000000000000000000000000 P c175dfccd77d4fcebebcdc0419cdd1d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c175dfccd77d4fcebebcdc0419cdd1d5, State: Initialized, Role: FOLLOWER
I20260925 07:29:59.411051 11252 consensus_queue.cc:260] T 00000000000000000000000000000000 P c175dfccd77d4fcebebcdc0419cdd1d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c175dfccd77d4fcebebcdc0419cdd1d5" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 45659 } }
I20260925 07:29:59.411144 11252 raft_consensus.cc:388] T 00000000000000000000000000000000 P c175dfccd77d4fcebebcdc0419cdd1d5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 07:29:59.411187 11252 raft_consensus.cc:482] T 00000000000000000000000000000000 P c175dfccd77d4fcebebcdc0419cdd1d5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 07:29:59.411232 11252 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c175dfccd77d4fcebebcdc0419cdd1d5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:29:59.411718 11252 raft_consensus.cc:504] T 00000000000000000000000000000000 P c175dfccd77d4fcebebcdc0419cdd1d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c175dfccd77d4fcebebcdc0419cdd1d5" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 45659 } }
I20260925 07:29:59.411825 11252 leader_election.cc:302] T 00000000000000000000000000000000 P c175dfccd77d4fcebebcdc0419cdd1d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c175dfccd77d4fcebebcdc0419cdd1d5; no voters:
I20260925 07:29:59.412021 11252 leader_election.cc:288] T 00000000000000000000000000000000 P c175dfccd77d4fcebebcdc0419cdd1d5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 07:29:59.412058 11257 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c175dfccd77d4fcebebcdc0419cdd1d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:29:59.412173 11257 raft_consensus.cc:686] T 00000000000000000000000000000000 P c175dfccd77d4fcebebcdc0419cdd1d5 [term 1 LEADER]: Becoming Leader. State: Replica: c175dfccd77d4fcebebcdc0419cdd1d5, State: Running, Role: LEADER
I20260925 07:29:59.412288 11252 sys_catalog.cc:564] T 00000000000000000000000000000000 P c175dfccd77d4fcebebcdc0419cdd1d5 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:29:59.412319 11257 consensus_queue.cc:237] T 00000000000000000000000000000000 P c175dfccd77d4fcebebcdc0419cdd1d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c175dfccd77d4fcebebcdc0419cdd1d5" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 45659 } }
I20260925 07:29:59.412775 11258 sys_catalog.cc:455] T 00000000000000000000000000000000 P c175dfccd77d4fcebebcdc0419cdd1d5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c175dfccd77d4fcebebcdc0419cdd1d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c175dfccd77d4fcebebcdc0419cdd1d5" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 45659 } } }
I20260925 07:29:59.412876 11258 sys_catalog.cc:458] T 00000000000000000000000000000000 P c175dfccd77d4fcebebcdc0419cdd1d5 [sys.catalog]: This master's current role is: LEADER
I20260925 07:29:59.413089 11259 sys_catalog.cc:455] T 00000000000000000000000000000000 P c175dfccd77d4fcebebcdc0419cdd1d5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c175dfccd77d4fcebebcdc0419cdd1d5. Latest consensus state: current_term: 1 leader_uuid: "c175dfccd77d4fcebebcdc0419cdd1d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c175dfccd77d4fcebebcdc0419cdd1d5" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 45659 } } }
I20260925 07:29:59.413370 11259 sys_catalog.cc:458] T 00000000000000000000000000000000 P c175dfccd77d4fcebebcdc0419cdd1d5 [sys.catalog]: This master's current role is: LEADER
W20260925 07:29:59.414074 11273 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c175dfccd77d4fcebebcdc0419cdd1d5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 07:29:59.414129 11273 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 07:29:59.414175 11265 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 07:29:59.414319 11265 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 25 07:29:59 dist-test-slave-57fn krb5kdc[11161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321399, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.66.62@KRBTEST.COM
I20260925 07:29:59.418143 11265 catalog_manager.cc:1347] Generated new cluster ID: 4f29f0b0e9014e0ea56adc7e2daed748
I20260925 07:29:59.418191 11265 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 07:29:59.420552 11256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:29:59.409546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46444 (local address 127.9.66.62:45659)
0925 07:29:59.409739 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0925 07:29:59.409748 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 07:29:59.409776 (+ 28us) server_negotiation.cc:407] Connection header received
0925 07:29:59.410693 (+ 917us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:29:59.410699 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:29:59.410744 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:29:59.410878 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:29:59.411570 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:29:59.412767 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:29:59.413900 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:29:59.414453 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:29:59.414582 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:29:59.417059 (+ 2477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:29:59.417063 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:29:59.417064 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:29:59.417075 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:29:59.418917 (+ 1842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:29:59.419289 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:29:59.419292 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:29:59.419292 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:29:59.419439 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:29:59.419645 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:29:59.419646 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:29:59.419647 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:29:59.419973 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:29:59.420135 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0925 07:29:59.420205 (+ 70us) server_negotiation.cc:299] Negotiation successful
0925 07:29:59.420347 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":80,"threads_started":1}
I20260925 07:29:59.425632 11265 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 07:29:59.426328 11265 catalog_manager.cc:1504] Loading token signing keys...
I20260925 07:29:59.431778 11265 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c175dfccd77d4fcebebcdc0419cdd1d5: Generated new TSK 0
I20260925 07:29:59.431936 11265 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.66.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.66.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.66.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.1.keytab.
Entry for principal kudu/127.9.66.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.1.keytab.
I20260925 07:29:59.456400 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790321392278483-9480-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790321392278483-9480-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790321392278483-9480-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790321392278483-9480-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.66.1:0
--local_ip_for_outbound_sockets=127.9.66.1
--webserver_interface=127.9.66.1
--webserver_port=0
--tserver_master_addrs=127.9.66.62:45659
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.1.keytab
--principal=kudu/127.9.66.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:37843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:29:59.545476 11278 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:29:59.545769 11278 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:29:59.545826 11278 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:29:59.546952 11278 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:29:59.547014 11278 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:29:59.547050 11278 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 07:29:59.547076 11278 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 07:29:59.547108 11278 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.66.1
I20260925 07:29:59.548542 11278 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:37843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790321392278483-9480-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790321392278483-9480-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.1.keytab
--principal=kudu/127.9.66.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.66.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790321392278483-9480-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.66.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.66.62:45659
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11278
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.66.1
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790321392278483-9480-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:29:59.548810 11278 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:29:59.549041 11278 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:29:59.554831 11283 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:29:59.554826 11284 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:29:59.554826 11286 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:29:59.555169 11278 server_base.cc:1034] running on GCE node
I20260925 07:29:59.555358 11278 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:29:59.555619 11278 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:29:59.556806 11278 hybrid_clock.cc:648] HybridClock initialized: now 1790321399556768 us; error 53 us; skew 500 ppm
Sep 25 07:29:59 dist-test-slave-57fn krb5kdc[11161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321399, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:29:59.559299 11278 init.cc:377] Logged in from keytab as kudu/127.9.66.1@KRBTEST.COM (short username kudu)
I20260925 07:29:59.559751 11278 webserver.cc:460] Webserver started at http://127.9.66.1:36039/ using document root <none> and password file <none>
I20260925 07:29:59.559984 11278 fs_manager.cc:359] Metadata directory not provided
I20260925 07:29:59.560051 11278 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:29:59.560197 11278 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:29:59.561265 11278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790321392278483-9480-0/minicluster-data/ts-0/data/instance:
uuid: "a429603a7ba54c9a80d82f953c7114c3"
format_stamp: "Formatted at 2026-09-25 07:29:59 on dist-test-slave-57fn"
server_key: "6da39b4a5ceaf56cead1e71e8b0e5458"
server_key_iv: "21acadd5924fbe40012fe9dffaaf2dfd"
server_key_version: "encryptionkey@0"
I20260925 07:29:59.561672 11278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790321392278483-9480-0/minicluster-data/ts-0/wal/instance:
uuid: "a429603a7ba54c9a80d82f953c7114c3"
format_stamp: "Formatted at 2026-09-25 07:29:59 on dist-test-slave-57fn"
server_key: "6da39b4a5ceaf56cead1e71e8b0e5458"
server_key_iv: "21acadd5924fbe40012fe9dffaaf2dfd"
server_key_version: "encryptionkey@0"
I20260925 07:29:59.563047 11278 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 07:29:59.563848 11293 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:29:59.564035 11278 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 07:29:59.564131 11278 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790321392278483-9480-0/minicluster-data/ts-0/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790321392278483-9480-0/minicluster-data/ts-0/wal
uuid: "a429603a7ba54c9a80d82f953c7114c3"
format_stamp: "Formatted at 2026-09-25 07:29:59 on dist-test-slave-57fn"
server_key: "6da39b4a5ceaf56cead1e71e8b0e5458"
server_key_iv: "21acadd5924fbe40012fe9dffaaf2dfd"
server_key_version: "encryptionkey@0"
I20260925 07:29:59.564215 11278 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790321392278483-9480-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790321392278483-9480-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790321392278483-9480-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:29:59.571260 11278 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:29:59.573618 11278 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:29:59.573769 11278 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:29:59.574406 11278 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:29:59.574491 11278 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:29:59.574533 11278 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:29:59.574563 11278 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:29:59.581987 11278 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.1:43749
I20260925 07:29:59.582091 11408 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.1:43749 every 8 connection(s)
Sep 25 07:29:59 dist-test-slave-57fn krb5kdc[11161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321399, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.1@KRBTEST.COM for kudu/127.9.66.62@KRBTEST.COM
I20260925 07:29:59.582408 11278 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790321392278483-9480-0/minicluster-data/ts-0/data/info.pb
I20260925 07:29:59.585242 11256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:29:59.574991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.1:48695 (local address 127.9.66.62:45659)
0925 07:29:59.575040 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 07:29:59.575044 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 07:29:59.575337 (+ 293us) server_negotiation.cc:407] Connection header received
0925 07:29:59.576046 (+ 709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:29:59.576047 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:29:59.576067 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:29:59.576119 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:29:59.577029 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:29:59.577714 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:29:59.579367 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:29:59.579588 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:29:59.579683 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:29:59.582875 (+ 3192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:29:59.582877 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:29:59.582877 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:29:59.582886 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:29:59.584250 (+ 1364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:29:59.584584 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:29:59.584584 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:29:59.584585 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:29:59.584632 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:29:59.584789 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:29:59.584790 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:29:59.584790 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:29:59.584879 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:29:59.584924 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0925 07:29:59.585121 (+ 197us) server_negotiation.cc:299] Negotiation successful
0925 07:29:59.585183 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 07:29:59.585413 11302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:29:59.574985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:45659 (local address 127.9.66.1:48695)
0925 07:29:59.575181 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0925 07:29:59.575228 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 07:29:59.575916 (+ 688us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:29:59.576203 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:29:59.576208 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:29:59.576617 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:29:59.576919 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:29:59.576925 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:29:59.577820 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:29:59.577822 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:29:59.579274 (+ 1452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:29:59.579277 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:29:59.579699 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:29:59.579701 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:29:59.579861 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:29:59.580903 (+ 1042us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:29:59.580912 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:29:59.582778 (+ 1866us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:29:59.584317 (+ 1539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:29:59.584321 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:29:59.584323 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:29:59.584520 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:29:59.584670 (+ 150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:29:59.584671 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:29:59.584671 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:29:59.584745 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:29:59.584930 (+ 185us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:29:59.584934 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:29:59.585035 (+ 101us) client_negotiation.cc:769] Sending connection context
0925 07:29:59.585110 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 07:29:59.585218 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":85,"threads_started":1}
I20260925 07:29:59.585963 11409 heartbeater.cc:347] Connected to a master server at 127.9.66.62:45659
I20260925 07:29:59.586042 11409 heartbeater.cc:464] Registering TS with master...
I20260925 07:29:59.586212 11409 heartbeater.cc:511] Master 127.9.66.62:45659 requested a full tablet report, sending...
I20260925 07:29:59.586778 11206 ts_manager.cc:194] Registered new tserver with Master: a429603a7ba54c9a80d82f953c7114c3 (127.9.66.1:43749)
I20260925 07:29:59.587705 11206 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.66.1@KRBTEST.COM'} at 127.9.66.1:48695
I20260925 07:29:59.591172 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 11278
I20260925 07:29:59.591288 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790321392278483-9480-0/minicluster-data/ts-0/wal/instance
I20260925 07:29:59.591485 9480 external_mini_cluster.cc:1428] Setting key 4789b16076c0df46c0fbcd34a1247e72
I20260925 07:29:59.592386 11256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:29:59.587052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.1:37605 (local address 127.9.66.62:45659)
0925 07:29:59.587099 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 07:29:59.587101 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:29:59.587173 (+ 72us) server_negotiation.cc:407] Connection header received
0925 07:29:59.587230 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:29:59.587230 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:29:59.587237 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:29:59.587279 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:29:59.587624 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:29:59.588252 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:29:59.588898 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:29:59.589064 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:29:59.589116 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:29:59.589886 (+ 770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:29:59.589888 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:29:59.589888 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:29:59.589893 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:29:59.591239 (+ 1346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:29:59.591641 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:29:59.591642 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:29:59.591642 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:29:59.591704 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:29:59.591920 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:29:59.591921 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:29:59.591921 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:29:59.592025 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:29:59.592103 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 07:29:59.592248 (+ 145us) server_negotiation.cc:299] Negotiation successful
0925 07:29:59.592320 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 07:29:59.592450 11415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:29:59.586965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:45659 (local address 127.9.66.1:37605)
0925 07:29:59.587111 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 07:29:59.587122 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 07:29:59.587179 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:29:59.587292 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:29:59.587293 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:29:59.587493 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:29:59.587561 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:29:59.587562 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:29:59.588322 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:29:59.588323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:29:59.588829 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:29:59.588831 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:29:59.589129 (+ 298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:29:59.589130 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:29:59.589184 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:29:59.589556 (+ 372us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:29:59.589561 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:29:59.589814 (+ 253us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:29:59.591328 (+ 1514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:29:59.591329 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:29:59.591329 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:29:59.591532 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:29:59.591772 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:29:59.591773 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:29:59.591773 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:29:59.591836 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:29:59.592108 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:29:59.592109 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:29:59.592149 (+ 40us) client_negotiation.cc:769] Sending connection context
0925 07:29:59.592214 (+ 65us) client_negotiation.cc:240] Negotiation successful
0925 07:29:59.592321 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":91,"threads_started":1}
WARNING: no policy specified for kudu/127.9.66.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.66.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.66.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.2.keytab.
Entry for principal kudu/127.9.66.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.2.keytab.
I20260925 07:29:59.616720 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790321392278483-9480-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790321392278483-9480-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790321392278483-9480-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790321392278483-9480-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.66.2:0
--local_ip_for_outbound_sockets=127.9.66.2
--webserver_interface=127.9.66.2
--webserver_port=0
--tserver_master_addrs=127.9.66.62:45659
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.2.keytab
--principal=kudu/127.9.66.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:37843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:29:59.693887 11418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:29:59.694109 11418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:29:59.694142 11418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:29:59.695252 11418 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:29:59.695317 11418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:29:59.695351 11418 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 07:29:59.695381 11418 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 07:29:59.695417 11418 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.66.2
I20260925 07:29:59.696794 11418 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:37843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790321392278483-9480-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790321392278483-9480-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.2.keytab
--principal=kudu/127.9.66.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.66.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790321392278483-9480-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.66.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.66.62:45659
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11418
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.66.2
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790321392278483-9480-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:29:59.697044 11418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:29:59.697259 11418 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:29:59.702049 11424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:29:59.701975 11426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:29:59.701980 11418 server_base.cc:1034] running on GCE node
W20260925 07:29:59.701995 11423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:29:59.702370 11418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:29:59.702626 11418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:29:59.703776 11418 hybrid_clock.cc:648] HybridClock initialized: now 1790321399703766 us; error 24 us; skew 500 ppm
Sep 25 07:29:59 dist-test-slave-57fn krb5kdc[11161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321399, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:29:59.706116 11418 init.cc:377] Logged in from keytab as kudu/127.9.66.2@KRBTEST.COM (short username kudu)
I20260925 07:29:59.706503 11418 webserver.cc:460] Webserver started at http://127.9.66.2:45391/ using document root <none> and password file <none>
I20260925 07:29:59.706722 11418 fs_manager.cc:359] Metadata directory not provided
I20260925 07:29:59.706806 11418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:29:59.706959 11418 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:29:59.707958 11418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790321392278483-9480-0/minicluster-data/ts-1/data/instance:
uuid: "d9c405e70885447fb3466fd8ceda0714"
format_stamp: "Formatted at 2026-09-25 07:29:59 on dist-test-slave-57fn"
server_key: "cdb8dfc36e8c220a81fb6e45ac0851be"
server_key_iv: "f85becaafafe791f9faae92fe6520ab9"
server_key_version: "encryptionkey@0"
I20260925 07:29:59.708302 11418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790321392278483-9480-0/minicluster-data/ts-1/wal/instance:
uuid: "d9c405e70885447fb3466fd8ceda0714"
format_stamp: "Formatted at 2026-09-25 07:29:59 on dist-test-slave-57fn"
server_key: "cdb8dfc36e8c220a81fb6e45ac0851be"
server_key_iv: "f85becaafafe791f9faae92fe6520ab9"
server_key_version: "encryptionkey@0"
I20260925 07:29:59.709620 11418 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 07:29:59.710374 11433 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:29:59.710584 11418 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:29:59.710687 11418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790321392278483-9480-0/minicluster-data/ts-1/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790321392278483-9480-0/minicluster-data/ts-1/wal
uuid: "d9c405e70885447fb3466fd8ceda0714"
format_stamp: "Formatted at 2026-09-25 07:29:59 on dist-test-slave-57fn"
server_key: "cdb8dfc36e8c220a81fb6e45ac0851be"
server_key_iv: "f85becaafafe791f9faae92fe6520ab9"
server_key_version: "encryptionkey@0"
I20260925 07:29:59.710754 11418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790321392278483-9480-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790321392278483-9480-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790321392278483-9480-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:29:59.725035 11418 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:29:59.726835 11418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:29:59.726969 11418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:29:59.727437 11418 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:29:59.727470 11418 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:29:59.727495 11418 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:29:59.727576 11418 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 07:29:59 dist-test-slave-57fn krb5kdc[11161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321399, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.2@KRBTEST.COM for kudu/127.9.66.62@KRBTEST.COM
I20260925 07:29:59.734625 11418 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.2:44051
I20260925 07:29:59.734711 11548 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.2:44051 every 8 connection(s)
I20260925 07:29:59.735090 11418 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790321392278483-9480-0/minicluster-data/ts-1/data/info.pb
I20260925 07:29:59.737195 11256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:29:59.727972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.2:47493 (local address 127.9.66.62:45659)
0925 07:29:59.728050 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0925 07:29:59.728053 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:29:59.728243 (+ 190us) server_negotiation.cc:407] Connection header received
0925 07:29:59.728705 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:29:59.728706 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:29:59.728719 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:29:59.728795 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:29:59.729740 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:29:59.730569 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:29:59.731701 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:29:59.732042 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:29:59.732115 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:29:59.734616 (+ 2501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:29:59.734618 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:29:59.734618 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:29:59.734625 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:29:59.736161 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:29:59.736515 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:29:59.736516 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:29:59.736516 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:29:59.736563 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:29:59.736724 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:29:59.736725 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:29:59.736725 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:29:59.736813 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:29:59.736856 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0925 07:29:59.737077 (+ 221us) server_negotiation.cc:299] Negotiation successful
0925 07:29:59.737134 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 07:29:59.737327 11442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:29:59.727943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:45659 (local address 127.9.66.2:47493)
0925 07:29:59.728114 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 07:29:59.728152 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 07:29:59.728597 (+ 445us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:29:59.728896 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:29:59.728900 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:29:59.729297 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:29:59.729633 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:29:59.729639 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:29:59.730701 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:29:59.730701 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:29:59.731596 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:29:59.731598 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:29:59.732142 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:29:59.732142 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:29:59.732276 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:29:59.732835 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:29:59.732843 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:29:59.734506 (+ 1663us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:29:59.736226 (+ 1720us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:29:59.736229 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:29:59.736231 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:29:59.736450 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:29:59.736602 (+ 152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:29:59.736603 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:29:59.736603 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:29:59.736680 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:29:59.736868 (+ 188us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:29:59.736872 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:29:59.736973 (+ 101us) client_negotiation.cc:769] Sending connection context
0925 07:29:59.737051 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 07:29:59.737158 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":108,"threads_started":1}
I20260925 07:29:59.737854 11549 heartbeater.cc:347] Connected to a master server at 127.9.66.62:45659
I20260925 07:29:59.737932 11549 heartbeater.cc:464] Registering TS with master...
I20260925 07:29:59.738103 11549 heartbeater.cc:511] Master 127.9.66.62:45659 requested a full tablet report, sending...
I20260925 07:29:59.738476 11205 ts_manager.cc:194] Registered new tserver with Master: d9c405e70885447fb3466fd8ceda0714 (127.9.66.2:44051)
I20260925 07:29:59.738909 11205 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.66.2@KRBTEST.COM'} at 127.9.66.2:47493
I20260925 07:29:59.741061 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 11418
I20260925 07:29:59.741169 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790321392278483-9480-0/minicluster-data/ts-1/wal/instance
I20260925 07:29:59.741364 9480 external_mini_cluster.cc:1428] Setting key e792f5e944a60820abd1446f86227b94
I20260925 07:29:59.744359 11555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:29:59.738810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:45659 (local address 127.9.66.2:40661)
0925 07:29:59.738985 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 07:29:59.738996 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 07:29:59.739125 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:29:59.739257 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:29:59.739258 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:29:59.739406 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:29:59.739486 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:29:59.739487 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:29:59.740170 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:29:59.740171 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:29:59.740702 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:29:59.740704 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:29:59.741026 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:29:59.741027 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:29:59.741108 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:29:59.741497 (+ 389us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:29:59.741502 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:29:59.741874 (+ 372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:29:59.743329 (+ 1455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:29:59.743330 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:29:59.743330 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:29:59.743555 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:29:59.743752 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:29:59.743752 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:29:59.743752 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:29:59.743796 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:29:59.744116 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:29:59.744117 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:29:59.744154 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 07:29:59.744218 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 07:29:59.744292 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":89,"threads_started":1}
I20260925 07:29:59.744437 11256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:29:59.738891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.2:40661 (local address 127.9.66.62:45659)
0925 07:29:59.738923 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0925 07:29:59.738925 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:29:59.739045 (+ 120us) server_negotiation.cc:407] Connection header received
0925 07:29:59.739186 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:29:59.739187 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:29:59.739194 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:29:59.739255 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:29:59.739557 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:29:59.740060 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:29:59.740771 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:29:59.740939 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:29:59.740988 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:29:59.741952 (+ 964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:29:59.741953 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:29:59.741953 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:29:59.741959 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:29:59.743250 (+ 1291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:29:59.743630 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:29:59.743631 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:29:59.743631 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:29:59.743689 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:29:59.743849 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:29:59.743850 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:29:59.743850 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:29:59.744031 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:29:59.744099 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 07:29:59.744258 (+ 159us) server_negotiation.cc:299] Negotiation successful
0925 07:29:59.744330 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13}
WARNING: no policy specified for kudu/127.9.66.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.66.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.66.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.3.keytab.
Entry for principal kudu/127.9.66.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.3.keytab.
I20260925 07:29:59.764540 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790321392278483-9480-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790321392278483-9480-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790321392278483-9480-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790321392278483-9480-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.66.3:0
--local_ip_for_outbound_sockets=127.9.66.3
--webserver_interface=127.9.66.3
--webserver_port=0
--tserver_master_addrs=127.9.66.62:45659
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.3.keytab
--principal=kudu/127.9.66.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:37843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:29:59.843634 11558 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:29:59.843816 11558 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:29:59.843839 11558 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:29:59.844849 11558 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:29:59.844879 11558 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:29:59.844898 11558 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 07:29:59.844910 11558 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 07:29:59.844926 11558 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.66.3
I20260925 07:29:59.846279 11558 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:37843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790321392278483-9480-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790321392278483-9480-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.3.keytab
--principal=kudu/127.9.66.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.66.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790321392278483-9480-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.66.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.66.62:45659
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11558
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.66.3
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790321392278483-9480-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:29:59.846459 11558 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:29:59.846603 11558 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 07:29:59.851445 11558 server_base.cc:1034] running on GCE node
W20260925 07:29:59.851379 11564 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:29:59.851395 11566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:29:59.851598 11563 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:29:59.851799 11558 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:29:59.852026 11558 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:29:59.853192 11558 hybrid_clock.cc:648] HybridClock initialized: now 1790321399853183 us; error 28 us; skew 500 ppm
Sep 25 07:29:59 dist-test-slave-57fn krb5kdc[11161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321399, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:29:59.855394 11558 init.cc:377] Logged in from keytab as kudu/127.9.66.3@KRBTEST.COM (short username kudu)
I20260925 07:29:59.855824 11558 webserver.cc:460] Webserver started at http://127.9.66.3:35945/ using document root <none> and password file <none>
I20260925 07:29:59.856045 11558 fs_manager.cc:359] Metadata directory not provided
I20260925 07:29:59.856109 11558 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:29:59.856248 11558 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:29:59.857208 11558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790321392278483-9480-0/minicluster-data/ts-2/data/instance:
uuid: "f713bc029900475cbefa33049b3bd9a3"
format_stamp: "Formatted at 2026-09-25 07:29:59 on dist-test-slave-57fn"
server_key: "d410b83414cee4aae417c6738d74db5f"
server_key_iv: "da8d39ec20e450a51cd0f55754570886"
server_key_version: "encryptionkey@0"
I20260925 07:29:59.857620 11558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790321392278483-9480-0/minicluster-data/ts-2/wal/instance:
uuid: "f713bc029900475cbefa33049b3bd9a3"
format_stamp: "Formatted at 2026-09-25 07:29:59 on dist-test-slave-57fn"
server_key: "d410b83414cee4aae417c6738d74db5f"
server_key_iv: "da8d39ec20e450a51cd0f55754570886"
server_key_version: "encryptionkey@0"
I20260925 07:29:59.858841 11558 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:29:59.859493 11573 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:29:59.859732 11558 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:29:59.859853 11558 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790321392278483-9480-0/minicluster-data/ts-2/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790321392278483-9480-0/minicluster-data/ts-2/wal
uuid: "f713bc029900475cbefa33049b3bd9a3"
format_stamp: "Formatted at 2026-09-25 07:29:59 on dist-test-slave-57fn"
server_key: "d410b83414cee4aae417c6738d74db5f"
server_key_iv: "da8d39ec20e450a51cd0f55754570886"
server_key_version: "encryptionkey@0"
I20260925 07:29:59.859939 11558 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790321392278483-9480-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790321392278483-9480-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790321392278483-9480-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:29:59.872702 11558 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:29:59.874444 11558 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:29:59.874596 11558 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:29:59.875051 11558 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:29:59.875111 11558 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:29:59.875152 11558 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:29:59.875178 11558 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 07:29:59 dist-test-slave-57fn krb5kdc[11161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321399, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.3@KRBTEST.COM for kudu/127.9.66.62@KRBTEST.COM
I20260925 07:29:59.882103 11558 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.3:37053
I20260925 07:29:59.882179 11688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.3:37053 every 8 connection(s)
I20260925 07:29:59.882493 11558 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790321392278483-9480-0/minicluster-data/ts-2/data/info.pb
I20260925 07:29:59.884702 11256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:29:59.875589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.3:49995 (local address 127.9.66.62:45659)
0925 07:29:59.875660 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 07:29:59.875663 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:29:59.875806 (+ 143us) server_negotiation.cc:407] Connection header received
0925 07:29:59.876283 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:29:59.876285 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:29:59.876297 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:29:59.876388 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:29:59.877146 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:29:59.877980 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:29:59.879275 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:29:59.879479 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:29:59.879562 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:29:59.882239 (+ 2677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:29:59.882240 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:29:59.882240 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:29:59.882249 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:29:59.883642 (+ 1393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:29:59.883991 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:29:59.883992 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:29:59.883992 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:29:59.884050 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:29:59.884259 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:29:59.884260 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:29:59.884260 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:29:59.884355 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:29:59.884409 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 07:29:59.884585 (+ 176us) server_negotiation.cc:299] Negotiation successful
0925 07:29:59.884642 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 07:29:59.884860 11582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:29:59.875544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:45659 (local address 127.9.66.3:49995)
0925 07:29:59.875687 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 07:29:59.875724 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 07:29:59.876178 (+ 454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:29:59.876477 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:29:59.876485 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:29:59.876784 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:29:59.877045 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:29:59.877051 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:29:59.878076 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:29:59.878077 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:29:59.879188 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:29:59.879190 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:29:59.879587 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:29:59.879588 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:29:59.879716 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:29:59.880210 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:29:59.880219 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:29:59.882154 (+ 1935us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:29:59.883708 (+ 1554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:29:59.883712 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:29:59.883714 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:29:59.883922 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:29:59.884116 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:29:59.884116 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:29:59.884116 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:29:59.884205 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:29:59.884401 (+ 196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:29:59.884403 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:29:59.884498 (+ 95us) client_negotiation.cc:769] Sending connection context
0925 07:29:59.884565 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 07:29:59.884670 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":82,"threads_started":1}
I20260925 07:29:59.885336 11689 heartbeater.cc:347] Connected to a master server at 127.9.66.62:45659
I20260925 07:29:59.885411 11689 heartbeater.cc:464] Registering TS with master...
I20260925 07:29:59.885612 11689 heartbeater.cc:511] Master 127.9.66.62:45659 requested a full tablet report, sending...
I20260925 07:29:59.885962 11206 ts_manager.cc:194] Registered new tserver with Master: f713bc029900475cbefa33049b3bd9a3 (127.9.66.3:37053)
I20260925 07:29:59.886451 11206 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.66.3@KRBTEST.COM'} at 127.9.66.3:49995
I20260925 07:29:59.889111 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 11558
I20260925 07:29:59.889240 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790321392278483-9480-0/minicluster-data/ts-2/wal/instance
I20260925 07:29:59.889447 9480 external_mini_cluster.cc:1428] Setting key fe3a921e3ee4ce80ce3dec59a75ef175
I20260925 07:29:59.890388 9480 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 07:29:59.892093 11256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:29:59.886378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.3:48949 (local address 127.9.66.62:45659)
0925 07:29:59.886416 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 07:29:59.886418 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:29:59.886547 (+ 129us) server_negotiation.cc:407] Connection header received
0925 07:29:59.886707 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:29:59.886708 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:29:59.886715 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:29:59.886788 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:29:59.887090 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:29:59.887687 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:29:59.888297 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:29:59.888473 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:29:59.888519 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:29:59.889349 (+ 830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:29:59.889350 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:29:59.889350 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:29:59.889356 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:29:59.890901 (+ 1545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:29:59.891255 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:29:59.891256 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:29:59.891256 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:29:59.891308 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:29:59.891513 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:29:59.891514 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:29:59.891515 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:29:59.891621 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:29:59.891695 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 07:29:59.891892 (+ 197us) server_negotiation.cc:299] Negotiation successful
0925 07:29:59.891979 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 07:29:59.892441 11695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:29:59.886285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:45659 (local address 127.9.66.3:48949)
0925 07:29:59.886471 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0925 07:29:59.886482 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 07:29:59.886640 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:29:59.886799 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:29:59.886800 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:29:59.886947 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:29:59.887012 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:29:59.887013 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:29:59.887752 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:29:59.887752 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:29:59.888232 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:29:59.888233 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:29:59.888532 (+ 299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:29:59.888532 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:29:59.888584 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:29:59.888911 (+ 327us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:29:59.888914 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:29:59.889197 (+ 283us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:29:59.891011 (+ 1814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:29:59.891012 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:29:59.891013 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:29:59.891180 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:29:59.891383 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:29:59.891384 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:29:59.891385 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:29:59.891441 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:29:59.891687 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:29:59.891688 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:29:59.891729 (+ 41us) client_negotiation.cc:769] Sending connection context
0925 07:29:59.892184 (+ 455us) client_negotiation.cc:240] Negotiation successful
0925 07:29:59.892306 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":83,"threads_started":1}
Sep 25 07:29:59 dist-test-slave-57fn krb5kdc[11161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321399, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 07:29:59 dist-test-slave-57fn krb5kdc[11161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321399, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.66.62@KRBTEST.COM
I20260925 07:29:59.911799 11256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:29:59.905713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46460 (local address 127.9.66.62:45659)
0925 07:29:59.905765 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 07:29:59.905767 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:29:59.905776 (+ 9us) server_negotiation.cc:407] Connection header received
0925 07:29:59.905801 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:29:59.905805 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:29:59.905814 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:29:59.905862 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:29:59.906357 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:29:59.906830 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:29:59.907497 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:29:59.907676 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:29:59.907725 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:29:59.909849 (+ 2124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:29:59.909850 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:29:59.909850 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:29:59.909857 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:29:59.910943 (+ 1086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:29:59.911233 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:29:59.911234 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:29:59.911234 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:29:59.911280 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:29:59.911449 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:29:59.911450 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:29:59.911450 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:29:59.911540 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:29:59.911610 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 07:29:59.911691 (+ 81us) server_negotiation.cc:299] Negotiation successful
0925 07:29:59.911741 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 07:29:59.913489 11206 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46460:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 07:29:59.915103 11206 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 07:29:59.921223 11703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:29:59.917159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.1:43749 (local address 127.0.0.1:46822)
0925 07:29:59.917425 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0925 07:29:59.917464 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 07:29:59.917685 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:29:59.917897 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:29:59.917900 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:29:59.917903 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:29:59.917967 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:29:59.917968 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:29:59.919277 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:29:59.919278 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:29:59.920316 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:29:59.920318 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:29:59.920975 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:29:59.920976 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:29:59.921035 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:29:59.921039 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 07:29:59.921067 (+ 28us) client_negotiation.cc:240] Negotiation successful
0925 07:29:59.921123 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"mutex_wait_us":3,"thread_start_us":51,"threads_started":1}
I20260925 07:29:59.921248 11708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:29:59.917433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43126 (local address 127.9.66.3:37053)
0925 07:29:59.917771 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0925 07:29:59.917775 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 07:29:59.917782 (+ 7us) server_negotiation.cc:407] Connection header received
0925 07:29:59.917810 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:29:59.917811 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:29:59.917822 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:29:59.918041 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:29:59.918098 (+ 57us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:29:59.918753 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:29:59.920084 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:29:59.920828 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:29:59.920905 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:29:59.920965 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 07:29:59.921115 (+ 150us) server_negotiation.cc:299] Negotiation successful
0925 07:29:59.921153 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":104,"threads_started":1}
I20260925 07:29:59.921223 11704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:29:59.917335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.3:37053 (local address 127.0.0.1:43126)
0925 07:29:59.917586 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0925 07:29:59.917595 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 07:29:59.917634 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:29:59.917938 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:29:59.917939 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:29:59.917940 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:29:59.917973 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:29:59.917974 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:29:59.918841 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:29:59.918842 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:29:59.919986 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:29:59.919989 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:29:59.920920 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:29:59.920921 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:29:59.921003 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:29:59.921017 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 07:29:59.921089 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 07:29:59.921122 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"mutex_wait_us":2,"thread_start_us":33,"threads_started":1}
I20260925 07:29:59.921558 11705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:29:59.917264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46822 (local address 127.9.66.1:43749)
0925 07:29:59.917433 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 07:29:59.917437 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 07:29:59.917582 (+ 145us) server_negotiation.cc:407] Connection header received
0925 07:29:59.917762 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:29:59.917765 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:29:59.917781 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:29:59.918075 (+ 294us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:29:59.918135 (+ 60us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:29:59.919185 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:29:59.920399 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:29:59.920862 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:29:59.921336 (+ 474us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:29:59.921395 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 07:29:59.921416 (+ 21us) server_negotiation.cc:299] Negotiation successful
0925 07:29:59.921443 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":74,"threads_started":1}
I20260925 07:29:59.922199 11343 tablet_service.cc:1467] Processing CreateTablet for tablet 8594719ec3db47189053b42f4e4244c4 (DEFAULT_TABLE table=test-table [id=9653d31c39474dd7abae8ab3bca03624]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:29:59.922314 11623 tablet_service.cc:1467] Processing CreateTablet for tablet 8594719ec3db47189053b42f4e4244c4 (DEFAULT_TABLE table=test-table [id=9653d31c39474dd7abae8ab3bca03624]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:29:59.922569 11343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8594719ec3db47189053b42f4e4244c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:29:59.922596 11623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8594719ec3db47189053b42f4e4244c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:29:59.922902 11707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:29:59.917159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.2:44051 (local address 127.0.0.1:57122)
0925 07:29:59.917436 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0925 07:29:59.917465 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0925 07:29:59.917585 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:29:59.917895 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:29:59.917898 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:29:59.917901 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:29:59.917979 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:29:59.917981 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:29:59.920403 (+ 2422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:29:59.920404 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:29:59.921766 (+ 1362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:29:59.921769 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:29:59.922674 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:29:59.922675 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:29:59.922734 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:29:59.922737 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 07:29:59.922804 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 07:29:59.922838 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"mutex_wait_us":5,"thread_start_us":52,"threads_started":1}
I20260925 07:29:59.922945 11706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:29:59.917264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57122 (local address 127.9.66.2:44051)
0925 07:29:59.917727 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0925 07:29:59.917732 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 07:29:59.917740 (+ 8us) server_negotiation.cc:407] Connection header received
0925 07:29:59.917778 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:29:59.917780 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:29:59.917793 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:29:59.918055 (+ 262us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:29:59.918116 (+ 61us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:29:59.919185 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:29:59.921857 (+ 2672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:29:59.922568 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:29:59.922664 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:29:59.922713 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 07:29:59.922819 (+ 106us) server_negotiation.cc:299] Negotiation successful
0925 07:29:59.922850 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":95,"threads_started":1}
I20260925 07:29:59.925824 11709 tablet_bootstrap.cc:492] T 8594719ec3db47189053b42f4e4244c4 P f713bc029900475cbefa33049b3bd9a3: Bootstrap starting.
I20260925 07:29:59.926040 11483 tablet_service.cc:1467] Processing CreateTablet for tablet 8594719ec3db47189053b42f4e4244c4 (DEFAULT_TABLE table=test-table [id=9653d31c39474dd7abae8ab3bca03624]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:29:59.926258 11483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8594719ec3db47189053b42f4e4244c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:29:59.926643 11709 tablet_bootstrap.cc:654] T 8594719ec3db47189053b42f4e4244c4 P f713bc029900475cbefa33049b3bd9a3: Neither blocks nor log segments found. Creating new log.
I20260925 07:29:59.926977 11709 log.cc:824] T 8594719ec3db47189053b42f4e4244c4 P f713bc029900475cbefa33049b3bd9a3: Log is configured to *not* fsync() on all Append() calls
I20260925 07:29:59.927656 11709 tablet_bootstrap.cc:492] T 8594719ec3db47189053b42f4e4244c4 P f713bc029900475cbefa33049b3bd9a3: No bootstrap required, opened a new log
I20260925 07:29:59.927739 11709 ts_tablet_manager.cc:1397] T 8594719ec3db47189053b42f4e4244c4 P f713bc029900475cbefa33049b3bd9a3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 07:29:59.929004 11712 tablet_bootstrap.cc:492] T 8594719ec3db47189053b42f4e4244c4 P d9c405e70885447fb3466fd8ceda0714: Bootstrap starting.
I20260925 07:29:59.929142 11710 tablet_bootstrap.cc:492] T 8594719ec3db47189053b42f4e4244c4 P a429603a7ba54c9a80d82f953c7114c3: Bootstrap starting.
I20260925 07:29:59.929693 11710 tablet_bootstrap.cc:654] T 8594719ec3db47189053b42f4e4244c4 P a429603a7ba54c9a80d82f953c7114c3: Neither blocks nor log segments found. Creating new log.
I20260925 07:29:59.929705 11712 tablet_bootstrap.cc:654] T 8594719ec3db47189053b42f4e4244c4 P d9c405e70885447fb3466fd8ceda0714: Neither blocks nor log segments found. Creating new log.
I20260925 07:29:59.929685 11709 raft_consensus.cc:348] T 8594719ec3db47189053b42f4e4244c4 P f713bc029900475cbefa33049b3bd9a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a429603a7ba54c9a80d82f953c7114c3" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 43749 } } peers { permanent_uuid: "d9c405e70885447fb3466fd8ceda0714" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 44051 } } peers { permanent_uuid: "f713bc029900475cbefa33049b3bd9a3" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 37053 } }
I20260925 07:29:59.929864 11709 raft_consensus.cc:374] T 8594719ec3db47189053b42f4e4244c4 P f713bc029900475cbefa33049b3bd9a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:29:59.929934 11709 raft_consensus.cc:729] T 8594719ec3db47189053b42f4e4244c4 P f713bc029900475cbefa33049b3bd9a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f713bc029900475cbefa33049b3bd9a3, State: Initialized, Role: FOLLOWER
I20260925 07:29:59.930034 11712 log.cc:824] T 8594719ec3db47189053b42f4e4244c4 P d9c405e70885447fb3466fd8ceda0714: Log is configured to *not* fsync() on all Append() calls
I20260925 07:29:59.930066 11710 log.cc:824] T 8594719ec3db47189053b42f4e4244c4 P a429603a7ba54c9a80d82f953c7114c3: Log is configured to *not* fsync() on all Append() calls
I20260925 07:29:59.930083 11709 consensus_queue.cc:260] T 8594719ec3db47189053b42f4e4244c4 P f713bc029900475cbefa33049b3bd9a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a429603a7ba54c9a80d82f953c7114c3" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 43749 } } peers { permanent_uuid: "d9c405e70885447fb3466fd8ceda0714" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 44051 } } peers { permanent_uuid: "f713bc029900475cbefa33049b3bd9a3" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 37053 } }
I20260925 07:29:59.930346 11709 ts_tablet_manager.cc:1428] T 8594719ec3db47189053b42f4e4244c4 P f713bc029900475cbefa33049b3bd9a3: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 07:29:59.930364 11689 heartbeater.cc:503] Master 127.9.66.62:45659 was elected leader, sending a full tablet report...
I20260925 07:29:59.930859 11710 tablet_bootstrap.cc:492] T 8594719ec3db47189053b42f4e4244c4 P a429603a7ba54c9a80d82f953c7114c3: No bootstrap required, opened a new log
I20260925 07:29:59.930949 11710 ts_tablet_manager.cc:1397] T 8594719ec3db47189053b42f4e4244c4 P a429603a7ba54c9a80d82f953c7114c3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 07:29:59.931041 11712 tablet_bootstrap.cc:492] T 8594719ec3db47189053b42f4e4244c4 P d9c405e70885447fb3466fd8ceda0714: No bootstrap required, opened a new log
I20260925 07:29:59.931111 11712 ts_tablet_manager.cc:1397] T 8594719ec3db47189053b42f4e4244c4 P d9c405e70885447fb3466fd8ceda0714: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 07:29:59.932619 11712 raft_consensus.cc:348] T 8594719ec3db47189053b42f4e4244c4 P d9c405e70885447fb3466fd8ceda0714 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a429603a7ba54c9a80d82f953c7114c3" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 43749 } } peers { permanent_uuid: "d9c405e70885447fb3466fd8ceda0714" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 44051 } } peers { permanent_uuid: "f713bc029900475cbefa33049b3bd9a3" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 37053 } }
I20260925 07:29:59.932777 11712 raft_consensus.cc:374] T 8594719ec3db47189053b42f4e4244c4 P d9c405e70885447fb3466fd8ceda0714 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:29:59.932824 11712 raft_consensus.cc:729] T 8594719ec3db47189053b42f4e4244c4 P d9c405e70885447fb3466fd8ceda0714 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9c405e70885447fb3466fd8ceda0714, State: Initialized, Role: FOLLOWER
I20260925 07:29:59.932760 11710 raft_consensus.cc:348] T 8594719ec3db47189053b42f4e4244c4 P a429603a7ba54c9a80d82f953c7114c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a429603a7ba54c9a80d82f953c7114c3" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 43749 } } peers { permanent_uuid: "d9c405e70885447fb3466fd8ceda0714" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 44051 } } peers { permanent_uuid: "f713bc029900475cbefa33049b3bd9a3" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 37053 } }
I20260925 07:29:59.932910 11710 raft_consensus.cc:374] T 8594719ec3db47189053b42f4e4244c4 P a429603a7ba54c9a80d82f953c7114c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:29:59.932953 11710 raft_consensus.cc:729] T 8594719ec3db47189053b42f4e4244c4 P a429603a7ba54c9a80d82f953c7114c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a429603a7ba54c9a80d82f953c7114c3, State: Initialized, Role: FOLLOWER
I20260925 07:29:59.932940 11712 consensus_queue.cc:260] T 8594719ec3db47189053b42f4e4244c4 P d9c405e70885447fb3466fd8ceda0714 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a429603a7ba54c9a80d82f953c7114c3" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 43749 } } peers { permanent_uuid: "d9c405e70885447fb3466fd8ceda0714" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 44051 } } peers { permanent_uuid: "f713bc029900475cbefa33049b3bd9a3" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 37053 } }
I20260925 07:29:59.933085 11710 consensus_queue.cc:260] T 8594719ec3db47189053b42f4e4244c4 P a429603a7ba54c9a80d82f953c7114c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a429603a7ba54c9a80d82f953c7114c3" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 43749 } } peers { permanent_uuid: "d9c405e70885447fb3466fd8ceda0714" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 44051 } } peers { permanent_uuid: "f713bc029900475cbefa33049b3bd9a3" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 37053 } }
I20260925 07:29:59.933179 11712 ts_tablet_manager.cc:1428] T 8594719ec3db47189053b42f4e4244c4 P d9c405e70885447fb3466fd8ceda0714: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 07:29:59.933188 11549 heartbeater.cc:503] Master 127.9.66.62:45659 was elected leader, sending a full tablet report...
I20260925 07:29:59.933377 11710 ts_tablet_manager.cc:1428] T 8594719ec3db47189053b42f4e4244c4 P a429603a7ba54c9a80d82f953c7114c3: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 07:29:59.933375 11409 heartbeater.cc:503] Master 127.9.66.62:45659 was elected leader, sending a full tablet report...
I20260925 07:29:59.942521 11716 raft_consensus.cc:482] T 8594719ec3db47189053b42f4e4244c4 P d9c405e70885447fb3466fd8ceda0714 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 07:29:59.942631 11716 raft_consensus.cc:504] T 8594719ec3db47189053b42f4e4244c4 P d9c405e70885447fb3466fd8ceda0714 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a429603a7ba54c9a80d82f953c7114c3" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 43749 } } peers { permanent_uuid: "d9c405e70885447fb3466fd8ceda0714" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 44051 } } peers { permanent_uuid: "f713bc029900475cbefa33049b3bd9a3" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 37053 } }
I20260925 07:29:59.942950 11716 leader_election.cc:288] T 8594719ec3db47189053b42f4e4244c4 P d9c405e70885447fb3466fd8ceda0714 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a429603a7ba54c9a80d82f953c7114c3 (127.9.66.1:43749), f713bc029900475cbefa33049b3bd9a3 (127.9.66.3:37053)
I20260925 07:29:59.946470 11708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:29:59.943207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.2:46929 (local address 127.9.66.3:37053)
0925 07:29:59.943251 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 07:29:59.943253 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:29:59.943269 (+ 16us) server_negotiation.cc:407] Connection header received
0925 07:29:59.943315 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:29:59.943315 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:29:59.943324 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:29:59.943371 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:29:59.943543 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:29:59.944143 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:29:59.945546 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:29:59.946094 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:29:59.946158 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:29:59.946168 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 07:29:59.946365 (+ 197us) server_negotiation.cc:299] Negotiation successful
0925 07:29:59.946403 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 07:29:59.946470 11705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:29:59.943214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.2:43855 (local address 127.9.66.1:43749)
0925 07:29:59.943260 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 07:29:59.943262 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:29:59.943563 (+ 301us) server_negotiation.cc:407] Connection header received
0925 07:29:59.943629 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:29:59.943629 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:29:59.943638 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:29:59.943674 (+ 36us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:29:59.943806 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:29:59.944291 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:29:59.945549 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:29:59.946082 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:29:59.946158 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:29:59.946168 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 07:29:59.946365 (+ 197us) server_negotiation.cc:299] Negotiation successful
0925 07:29:59.946407 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 07:29:59.946473 11442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:29:59.943140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.3:37053 (local address 127.9.66.2:46929)
0925 07:29:59.943185 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0925 07:29:59.943196 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 07:29:59.943245 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:29:59.943396 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:29:59.943397 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:29:59.943399 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:29:59.943465 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:29:59.943467 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:29:59.944244 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:29:59.944245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:29:59.945438 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:29:59.945440 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:29:59.946190 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:29:59.946191 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:29:59.946279 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:29:59.946281 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 07:29:59.946358 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 07:29:59.946390 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27,"mutex_wait_us":2}
I20260925 07:29:59.946483 11718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:29:59.943141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.1:43749 (local address 127.9.66.2:43855)
0925 07:29:59.943494 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0925 07:29:59.943519 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 07:29:59.943575 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:29:59.943677 (+ 102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:29:59.943678 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:29:59.943679 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:29:59.943746 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:29:59.943747 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:29:59.944377 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:29:59.944378 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:29:59.945453 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:29:59.945454 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:29:59.946190 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:29:59.946191 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:29:59.946277 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:29:59.946282 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 07:29:59.946349 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 07:29:59.946389 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"mutex_wait_us":180,"thread_start_us":106,"threads_started":1}
I20260925 07:29:59.946861 11363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8594719ec3db47189053b42f4e4244c4" candidate_uuid: "d9c405e70885447fb3466fd8ceda0714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a429603a7ba54c9a80d82f953c7114c3" is_pre_election: true
I20260925 07:29:59.946888 11643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8594719ec3db47189053b42f4e4244c4" candidate_uuid: "d9c405e70885447fb3466fd8ceda0714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f713bc029900475cbefa33049b3bd9a3" is_pre_election: true
I20260925 07:29:59.947010 11363 raft_consensus.cc:2454] T 8594719ec3db47189053b42f4e4244c4 P a429603a7ba54c9a80d82f953c7114c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9c405e70885447fb3466fd8ceda0714 in term 0.
I20260925 07:29:59.947010 11643 raft_consensus.cc:2454] T 8594719ec3db47189053b42f4e4244c4 P f713bc029900475cbefa33049b3bd9a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9c405e70885447fb3466fd8ceda0714 in term 0.
I20260925 07:29:59.947213 11435 leader_election.cc:302] T 8594719ec3db47189053b42f4e4244c4 P d9c405e70885447fb3466fd8ceda0714 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d9c405e70885447fb3466fd8ceda0714, f713bc029900475cbefa33049b3bd9a3; no voters:
I20260925 07:29:59.947321 11716 raft_consensus.cc:2781] T 8594719ec3db47189053b42f4e4244c4 P d9c405e70885447fb3466fd8ceda0714 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 07:29:59.947383 11716 raft_consensus.cc:482] T 8594719ec3db47189053b42f4e4244c4 P d9c405e70885447fb3466fd8ceda0714 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 07:29:59.947417 11716 raft_consensus.cc:3037] T 8594719ec3db47189053b42f4e4244c4 P d9c405e70885447fb3466fd8ceda0714 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:29:59.948064 11716 raft_consensus.cc:504] T 8594719ec3db47189053b42f4e4244c4 P d9c405e70885447fb3466fd8ceda0714 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a429603a7ba54c9a80d82f953c7114c3" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 43749 } } peers { permanent_uuid: "d9c405e70885447fb3466fd8ceda0714" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 44051 } } peers { permanent_uuid: "f713bc029900475cbefa33049b3bd9a3" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 37053 } }
I20260925 07:29:59.948199 11716 leader_election.cc:288] T 8594719ec3db47189053b42f4e4244c4 P d9c405e70885447fb3466fd8ceda0714 [CANDIDATE]: Term 1 election: Requested vote from peers a429603a7ba54c9a80d82f953c7114c3 (127.9.66.1:43749), f713bc029900475cbefa33049b3bd9a3 (127.9.66.3:37053)
I20260925 07:29:59.948335 11643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8594719ec3db47189053b42f4e4244c4" candidate_uuid: "d9c405e70885447fb3466fd8ceda0714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f713bc029900475cbefa33049b3bd9a3"
I20260925 07:29:59.948335 11363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8594719ec3db47189053b42f4e4244c4" candidate_uuid: "d9c405e70885447fb3466fd8ceda0714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a429603a7ba54c9a80d82f953c7114c3"
I20260925 07:29:59.948407 11363 raft_consensus.cc:3037] T 8594719ec3db47189053b42f4e4244c4 P a429603a7ba54c9a80d82f953c7114c3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:29:59.948407 11643 raft_consensus.cc:3037] T 8594719ec3db47189053b42f4e4244c4 P f713bc029900475cbefa33049b3bd9a3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:29:59.949188 11643 raft_consensus.cc:2454] T 8594719ec3db47189053b42f4e4244c4 P f713bc029900475cbefa33049b3bd9a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9c405e70885447fb3466fd8ceda0714 in term 1.
I20260925 07:29:59.949188 11363 raft_consensus.cc:2454] T 8594719ec3db47189053b42f4e4244c4 P a429603a7ba54c9a80d82f953c7114c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9c405e70885447fb3466fd8ceda0714 in term 1.
I20260925 07:29:59.949343 11435 leader_election.cc:302] T 8594719ec3db47189053b42f4e4244c4 P d9c405e70885447fb3466fd8ceda0714 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d9c405e70885447fb3466fd8ceda0714, f713bc029900475cbefa33049b3bd9a3; no voters:
I20260925 07:29:59.949440 11716 raft_consensus.cc:2781] T 8594719ec3db47189053b42f4e4244c4 P d9c405e70885447fb3466fd8ceda0714 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:29:59.949580 11716 raft_consensus.cc:686] T 8594719ec3db47189053b42f4e4244c4 P d9c405e70885447fb3466fd8ceda0714 [term 1 LEADER]: Becoming Leader. State: Replica: d9c405e70885447fb3466fd8ceda0714, State: Running, Role: LEADER
I20260925 07:29:59.949678 11716 consensus_queue.cc:237] T 8594719ec3db47189053b42f4e4244c4 P d9c405e70885447fb3466fd8ceda0714 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a429603a7ba54c9a80d82f953c7114c3" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 43749 } } peers { permanent_uuid: "d9c405e70885447fb3466fd8ceda0714" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 44051 } } peers { permanent_uuid: "f713bc029900475cbefa33049b3bd9a3" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 37053 } }
I20260925 07:29:59.950424 11206 catalog_manager.cc:5495] T 8594719ec3db47189053b42f4e4244c4 P d9c405e70885447fb3466fd8ceda0714 reported cstate change: term changed from 0 to 1, leader changed from <none> to d9c405e70885447fb3466fd8ceda0714 (127.9.66.2). New cstate: current_term: 1 leader_uuid: "d9c405e70885447fb3466fd8ceda0714" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a429603a7ba54c9a80d82f953c7114c3" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 43749 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d9c405e70885447fb3466fd8ceda0714" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 44051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f713bc029900475cbefa33049b3bd9a3" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 37053 } health_report { overall_health: UNKNOWN } } }
Sep 25 07:29:59 dist-test-slave-57fn krb5kdc[11161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321399, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.62@KRBTEST.COM for kudu/127.9.66.62@KRBTEST.COM
I20260925 07:29:59.980384 11726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:29:59.974243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:45659 (local address 127.0.0.1:46474)
0925 07:29:59.974420 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0925 07:29:59.974429 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 07:29:59.974482 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:29:59.974640 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:29:59.974640 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:29:59.974902 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:29:59.974942 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:29:59.974944 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:29:59.975524 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:29:59.975525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:29:59.976013 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:29:59.976014 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:29:59.976298 (+ 284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:29:59.976299 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:29:59.976335 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:29:59.976841 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:29:59.976845 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:29:59.978176 (+ 1331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:29:59.979485 (+ 1309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:29:59.979491 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:29:59.979493 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:29:59.979671 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:29:59.979860 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:29:59.979861 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:29:59.979861 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:29:59.979915 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:29:59.980131 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:29:59.980137 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:29:59.980176 (+ 39us) client_negotiation.cc:769] Sending connection context
0925 07:29:59.980227 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 07:29:59.980309 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":105,"threads_started":1}
I20260925 07:29:59.980439 11256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:29:59.974293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46474 (local address 127.9.66.62:45659)
0925 07:29:59.974373 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0925 07:29:59.974375 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:29:59.974490 (+ 115us) server_negotiation.cc:407] Connection header received
0925 07:29:59.974524 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:29:59.974525 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:29:59.974533 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:29:59.974596 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:29:59.975006 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:29:59.975456 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:29:59.976078 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:29:59.976241 (+ 163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:29:59.976285 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:29:59.978247 (+ 1962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:29:59.978248 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:29:59.978249 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:29:59.978254 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:29:59.979420 (+ 1166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:29:59.979749 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:29:59.979750 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:29:59.979751 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:29:59.979807 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:29:59.979993 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:29:59.979994 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:29:59.979994 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:29:59.980081 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:29:59.980127 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0925 07:29:59.980256 (+ 129us) server_negotiation.cc:299] Negotiation successful
0925 07:29:59.980360 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":39,"server-negotiator.queue_time_us":42}
I20260925 07:29:59.982497 11206 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.9.66.62@KRBTEST.COM'} at 127.0.0.1:46474:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 07:29:59.982685 11206 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 07:29:59.984038 11343 tablet_service.cc:1467] Processing CreateTablet for tablet 35d2ed81f27f4d87bbf02db370e612f1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8f315d4ebad44fb8801bed5fff8f884a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 07:29:59.984205 11343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35d2ed81f27f4d87bbf02db370e612f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:29:59.984197 11483 tablet_service.cc:1467] Processing CreateTablet for tablet 35d2ed81f27f4d87bbf02db370e612f1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8f315d4ebad44fb8801bed5fff8f884a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 07:29:59.984347 11483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35d2ed81f27f4d87bbf02db370e612f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:29:59.985561 11710 tablet_bootstrap.cc:492] T 35d2ed81f27f4d87bbf02db370e612f1 P a429603a7ba54c9a80d82f953c7114c3: Bootstrap starting.
W20260925 07:29:59.985649 11550 tablet.cc:2369] T 8594719ec3db47189053b42f4e4244c4 P d9c405e70885447fb3466fd8ceda0714: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 07:29:59.985795 11712 tablet_bootstrap.cc:492] T 35d2ed81f27f4d87bbf02db370e612f1 P d9c405e70885447fb3466fd8ceda0714: Bootstrap starting.
I20260925 07:29:59.985926 11623 tablet_service.cc:1467] Processing CreateTablet for tablet 35d2ed81f27f4d87bbf02db370e612f1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8f315d4ebad44fb8801bed5fff8f884a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 07:29:59.986111 11623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35d2ed81f27f4d87bbf02db370e612f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:29:59.986207 11712 tablet_bootstrap.cc:654] T 35d2ed81f27f4d87bbf02db370e612f1 P d9c405e70885447fb3466fd8ceda0714: Neither blocks nor log segments found. Creating new log.
I20260925 07:29:59.986307 11710 tablet_bootstrap.cc:654] T 35d2ed81f27f4d87bbf02db370e612f1 P a429603a7ba54c9a80d82f953c7114c3: Neither blocks nor log segments found. Creating new log.
I20260925 07:29:59.986971 11710 tablet_bootstrap.cc:492] T 35d2ed81f27f4d87bbf02db370e612f1 P a429603a7ba54c9a80d82f953c7114c3: No bootstrap required, opened a new log
I20260925 07:29:59.987030 11710 ts_tablet_manager.cc:1397] T 35d2ed81f27f4d87bbf02db370e612f1 P a429603a7ba54c9a80d82f953c7114c3: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 07:29:59.987193 11710 raft_consensus.cc:348] T 35d2ed81f27f4d87bbf02db370e612f1 P a429603a7ba54c9a80d82f953c7114c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a429603a7ba54c9a80d82f953c7114c3" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 43749 } } peers { permanent_uuid: "d9c405e70885447fb3466fd8ceda0714" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 44051 } } peers { permanent_uuid: "f713bc029900475cbefa33049b3bd9a3" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 37053 } }
I20260925 07:29:59.987265 11710 raft_consensus.cc:374] T 35d2ed81f27f4d87bbf02db370e612f1 P a429603a7ba54c9a80d82f953c7114c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:29:59.987294 11710 raft_consensus.cc:729] T 35d2ed81f27f4d87bbf02db370e612f1 P a429603a7ba54c9a80d82f953c7114c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a429603a7ba54c9a80d82f953c7114c3, State: Initialized, Role: FOLLOWER
I20260925 07:29:59.987360 11710 consensus_queue.cc:260] T 35d2ed81f27f4d87bbf02db370e612f1 P a429603a7ba54c9a80d82f953c7114c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a429603a7ba54c9a80d82f953c7114c3" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 43749 } } peers { permanent_uuid: "d9c405e70885447fb3466fd8ceda0714" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 44051 } } peers { permanent_uuid: "f713bc029900475cbefa33049b3bd9a3" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 37053 } }
I20260925 07:29:59.987458 11710 ts_tablet_manager.cc:1428] T 35d2ed81f27f4d87bbf02db370e612f1 P a429603a7ba54c9a80d82f953c7114c3: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 07:29:59.987536 11717 tablet_replica.cc:439] T 35d2ed81f27f4d87bbf02db370e612f1 P a429603a7ba54c9a80d82f953c7114c3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a429603a7ba54c9a80d82f953c7114c3" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 43749 } } peers { permanent_uuid: "d9c405e70885447fb3466fd8ceda0714" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 44051 } } peers { permanent_uuid: "f713bc029900475cbefa33049b3bd9a3" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 37053 } } }
I20260925 07:29:59.987623 11709 tablet_bootstrap.cc:492] T 35d2ed81f27f4d87bbf02db370e612f1 P f713bc029900475cbefa33049b3bd9a3: Bootstrap starting.
I20260925 07:29:59.987632 11717 tablet_replica.cc:442] T 35d2ed81f27f4d87bbf02db370e612f1 P a429603a7ba54c9a80d82f953c7114c3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 07:29:59.988128 11712 tablet_bootstrap.cc:492] T 35d2ed81f27f4d87bbf02db370e612f1 P d9c405e70885447fb3466fd8ceda0714: No bootstrap required, opened a new log
I20260925 07:29:59.988160 11709 tablet_bootstrap.cc:654] T 35d2ed81f27f4d87bbf02db370e612f1 P f713bc029900475cbefa33049b3bd9a3: Neither blocks nor log segments found. Creating new log.
I20260925 07:29:59.988184 11712 ts_tablet_manager.cc:1397] T 35d2ed81f27f4d87bbf02db370e612f1 P d9c405e70885447fb3466fd8ceda0714: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 07:29:59.988327 11712 raft_consensus.cc:348] T 35d2ed81f27f4d87bbf02db370e612f1 P d9c405e70885447fb3466fd8ceda0714 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a429603a7ba54c9a80d82f953c7114c3" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 43749 } } peers { permanent_uuid: "d9c405e70885447fb3466fd8ceda0714" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 44051 } } peers { permanent_uuid: "f713bc029900475cbefa33049b3bd9a3" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 37053 } }
I20260925 07:29:59.988417 11712 raft_consensus.cc:374] T 35d2ed81f27f4d87bbf02db370e612f1 P d9c405e70885447fb3466fd8ceda0714 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:29:59.988449 11712 raft_consensus.cc:729] T 35d2ed81f27f4d87bbf02db370e612f1 P d9c405e70885447fb3466fd8ceda0714 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9c405e70885447fb3466fd8ceda0714, State: Initialized, Role: FOLLOWER
I20260925 07:29:59.988505 11712 consensus_queue.cc:260] T 35d2ed81f27f4d87bbf02db370e612f1 P d9c405e70885447fb3466fd8ceda0714 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a429603a7ba54c9a80d82f953c7114c3" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 43749 } } peers { permanent_uuid: "d9c405e70885447fb3466fd8ceda0714" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 44051 } } peers { permanent_uuid: "f713bc029900475cbefa33049b3bd9a3" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 37053 } }
I20260925 07:29:59.988600 11712 ts_tablet_manager.cc:1428] T 35d2ed81f27f4d87bbf02db370e612f1 P d9c405e70885447fb3466fd8ceda0714: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 07:29:59.988629 11716 tablet_replica.cc:439] T 35d2ed81f27f4d87bbf02db370e612f1 P d9c405e70885447fb3466fd8ceda0714: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a429603a7ba54c9a80d82f953c7114c3" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 43749 } } peers { permanent_uuid: "d9c405e70885447fb3466fd8ceda0714" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 44051 } } peers { permanent_uuid: "f713bc029900475cbefa33049b3bd9a3" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 37053 } } }
I20260925 07:29:59.988699 11716 tablet_replica.cc:442] T 35d2ed81f27f4d87bbf02db370e612f1 P d9c405e70885447fb3466fd8ceda0714: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 07:29:59.988842 11709 tablet_bootstrap.cc:492] T 35d2ed81f27f4d87bbf02db370e612f1 P f713bc029900475cbefa33049b3bd9a3: No bootstrap required, opened a new log
I20260925 07:29:59.988895 11709 ts_tablet_manager.cc:1397] T 35d2ed81f27f4d87bbf02db370e612f1 P f713bc029900475cbefa33049b3bd9a3: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 07:29:59.989019 11709 raft_consensus.cc:348] T 35d2ed81f27f4d87bbf02db370e612f1 P f713bc029900475cbefa33049b3bd9a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a429603a7ba54c9a80d82f953c7114c3" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 43749 } } peers { permanent_uuid: "d9c405e70885447fb3466fd8ceda0714" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 44051 } } peers { permanent_uuid: "f713bc029900475cbefa33049b3bd9a3" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 37053 } }
I20260925 07:29:59.989060 11709 raft_consensus.cc:374] T 35d2ed81f27f4d87bbf02db370e612f1 P f713bc029900475cbefa33049b3bd9a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:29:59.989073 11709 raft_consensus.cc:729] T 35d2ed81f27f4d87bbf02db370e612f1 P f713bc029900475cbefa33049b3bd9a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f713bc029900475cbefa33049b3bd9a3, State: Initialized, Role: FOLLOWER
I20260925 07:29:59.989127 11709 consensus_queue.cc:260] T 35d2ed81f27f4d87bbf02db370e612f1 P f713bc029900475cbefa33049b3bd9a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a429603a7ba54c9a80d82f953c7114c3" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 43749 } } peers { permanent_uuid: "d9c405e70885447fb3466fd8ceda0714" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 44051 } } peers { permanent_uuid: "f713bc029900475cbefa33049b3bd9a3" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 37053 } }
I20260925 07:29:59.989212 11709 ts_tablet_manager.cc:1428] T 35d2ed81f27f4d87bbf02db370e612f1 P f713bc029900475cbefa33049b3bd9a3: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 07:29:59.989261 11714 tablet_replica.cc:439] T 35d2ed81f27f4d87bbf02db370e612f1 P f713bc029900475cbefa33049b3bd9a3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a429603a7ba54c9a80d82f953c7114c3" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 43749 } } peers { permanent_uuid: "d9c405e70885447fb3466fd8ceda0714" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 44051 } } peers { permanent_uuid: "f713bc029900475cbefa33049b3bd9a3" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 37053 } } }
I20260925 07:29:59.989401 11714 tablet_replica.cc:442] T 35d2ed81f27f4d87bbf02db370e612f1 P f713bc029900475cbefa33049b3bd9a3: This TxnStatusTablet replica's current role is: FOLLOWER
W20260925 07:30:00.083137 11410 tablet.cc:2369] T 35d2ed81f27f4d87bbf02db370e612f1 P a429603a7ba54c9a80d82f953c7114c3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 07:30:00.133054 11690 tablet.cc:2369] T 35d2ed81f27f4d87bbf02db370e612f1 P f713bc029900475cbefa33049b3bd9a3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 07:30:00.165293 11716 raft_consensus.cc:482] T 35d2ed81f27f4d87bbf02db370e612f1 P d9c405e70885447fb3466fd8ceda0714 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 07:30:00.165393 11716 raft_consensus.cc:504] T 35d2ed81f27f4d87bbf02db370e612f1 P d9c405e70885447fb3466fd8ceda0714 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a429603a7ba54c9a80d82f953c7114c3" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 43749 } } peers { permanent_uuid: "d9c405e70885447fb3466fd8ceda0714" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 44051 } } peers { permanent_uuid: "f713bc029900475cbefa33049b3bd9a3" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 37053 } }
I20260925 07:30:00.165570 11716 leader_election.cc:288] T 35d2ed81f27f4d87bbf02db370e612f1 P d9c405e70885447fb3466fd8ceda0714 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a429603a7ba54c9a80d82f953c7114c3 (127.9.66.1:43749), f713bc029900475cbefa33049b3bd9a3 (127.9.66.3:37053)
I20260925 07:30:00.165786 11363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35d2ed81f27f4d87bbf02db370e612f1" candidate_uuid: "d9c405e70885447fb3466fd8ceda0714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a429603a7ba54c9a80d82f953c7114c3" is_pre_election: true
I20260925 07:30:00.165786 11643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35d2ed81f27f4d87bbf02db370e612f1" candidate_uuid: "d9c405e70885447fb3466fd8ceda0714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f713bc029900475cbefa33049b3bd9a3" is_pre_election: true
I20260925 07:30:00.165891 11363 raft_consensus.cc:2454] T 35d2ed81f27f4d87bbf02db370e612f1 P a429603a7ba54c9a80d82f953c7114c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9c405e70885447fb3466fd8ceda0714 in term 0.
I20260925 07:30:00.165891 11643 raft_consensus.cc:2454] T 35d2ed81f27f4d87bbf02db370e612f1 P f713bc029900475cbefa33049b3bd9a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9c405e70885447fb3466fd8ceda0714 in term 0.
I20260925 07:30:00.166036 11435 leader_election.cc:302] T 35d2ed81f27f4d87bbf02db370e612f1 P d9c405e70885447fb3466fd8ceda0714 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d9c405e70885447fb3466fd8ceda0714, f713bc029900475cbefa33049b3bd9a3; no voters:
I20260925 07:30:00.166158 11716 raft_consensus.cc:2781] T 35d2ed81f27f4d87bbf02db370e612f1 P d9c405e70885447fb3466fd8ceda0714 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 07:30:00.166199 11716 raft_consensus.cc:482] T 35d2ed81f27f4d87bbf02db370e612f1 P d9c405e70885447fb3466fd8ceda0714 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 07:30:00.166213 11716 raft_consensus.cc:3037] T 35d2ed81f27f4d87bbf02db370e612f1 P d9c405e70885447fb3466fd8ceda0714 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:30:00.166966 11716 raft_consensus.cc:504] T 35d2ed81f27f4d87bbf02db370e612f1 P d9c405e70885447fb3466fd8ceda0714 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a429603a7ba54c9a80d82f953c7114c3" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 43749 } } peers { permanent_uuid: "d9c405e70885447fb3466fd8ceda0714" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 44051 } } peers { permanent_uuid: "f713bc029900475cbefa33049b3bd9a3" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 37053 } }
I20260925 07:30:00.167110 11716 leader_election.cc:288] T 35d2ed81f27f4d87bbf02db370e612f1 P d9c405e70885447fb3466fd8ceda0714 [CANDIDATE]: Term 1 election: Requested vote from peers a429603a7ba54c9a80d82f953c7114c3 (127.9.66.1:43749), f713bc029900475cbefa33049b3bd9a3 (127.9.66.3:37053)
I20260925 07:30:00.167292 11363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35d2ed81f27f4d87bbf02db370e612f1" candidate_uuid: "d9c405e70885447fb3466fd8ceda0714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a429603a7ba54c9a80d82f953c7114c3"
I20260925 07:30:00.167300 11643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35d2ed81f27f4d87bbf02db370e612f1" candidate_uuid: "d9c405e70885447fb3466fd8ceda0714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f713bc029900475cbefa33049b3bd9a3"
I20260925 07:30:00.167372 11643 raft_consensus.cc:3037] T 35d2ed81f27f4d87bbf02db370e612f1 P f713bc029900475cbefa33049b3bd9a3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:30:00.167388 11363 raft_consensus.cc:3037] T 35d2ed81f27f4d87bbf02db370e612f1 P a429603a7ba54c9a80d82f953c7114c3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:30:00.168049 11363 raft_consensus.cc:2454] T 35d2ed81f27f4d87bbf02db370e612f1 P a429603a7ba54c9a80d82f953c7114c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9c405e70885447fb3466fd8ceda0714 in term 1.
I20260925 07:30:00.168102 11643 raft_consensus.cc:2454] T 35d2ed81f27f4d87bbf02db370e612f1 P f713bc029900475cbefa33049b3bd9a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9c405e70885447fb3466fd8ceda0714 in term 1.
I20260925 07:30:00.168244 11434 leader_election.cc:302] T 35d2ed81f27f4d87bbf02db370e612f1 P d9c405e70885447fb3466fd8ceda0714 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a429603a7ba54c9a80d82f953c7114c3, d9c405e70885447fb3466fd8ceda0714; no voters:
I20260925 07:30:00.168373 11716 raft_consensus.cc:2781] T 35d2ed81f27f4d87bbf02db370e612f1 P d9c405e70885447fb3466fd8ceda0714 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:30:00.168426 11716 raft_consensus.cc:686] T 35d2ed81f27f4d87bbf02db370e612f1 P d9c405e70885447fb3466fd8ceda0714 [term 1 LEADER]: Becoming Leader. State: Replica: d9c405e70885447fb3466fd8ceda0714, State: Running, Role: LEADER
I20260925 07:30:00.168463 11716 consensus_queue.cc:237] T 35d2ed81f27f4d87bbf02db370e612f1 P d9c405e70885447fb3466fd8ceda0714 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a429603a7ba54c9a80d82f953c7114c3" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 43749 } } peers { permanent_uuid: "d9c405e70885447fb3466fd8ceda0714" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 44051 } } peers { permanent_uuid: "f713bc029900475cbefa33049b3bd9a3" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 37053 } }
I20260925 07:30:00.168808 11719 tablet_replica.cc:439] T 35d2ed81f27f4d87bbf02db370e612f1 P d9c405e70885447fb3466fd8ceda0714: TxnStatusTablet state changed. Reason: New leader d9c405e70885447fb3466fd8ceda0714. Latest consensus state: current_term: 1 leader_uuid: "d9c405e70885447fb3466fd8ceda0714" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a429603a7ba54c9a80d82f953c7114c3" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 43749 } } peers { permanent_uuid: "d9c405e70885447fb3466fd8ceda0714" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 44051 } } peers { permanent_uuid: "f713bc029900475cbefa33049b3bd9a3" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 37053 } } }
I20260925 07:30:00.168895 11719 tablet_replica.cc:442] T 35d2ed81f27f4d87bbf02db370e612f1 P d9c405e70885447fb3466fd8ceda0714: This TxnStatusTablet replica's current role is: LEADER
I20260925 07:30:00.169093 11731 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 07:30:00.169100 11205 catalog_manager.cc:5495] T 35d2ed81f27f4d87bbf02db370e612f1 P d9c405e70885447fb3466fd8ceda0714 reported cstate change: term changed from 0 to 1, leader changed from <none> to d9c405e70885447fb3466fd8ceda0714 (127.9.66.2). New cstate: current_term: 1 leader_uuid: "d9c405e70885447fb3466fd8ceda0714" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a429603a7ba54c9a80d82f953c7114c3" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 43749 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d9c405e70885447fb3466fd8ceda0714" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 44051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f713bc029900475cbefa33049b3bd9a3" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 37053 } health_report { overall_health: UNKNOWN } } }
I20260925 07:30:00.169227 11731 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 07:30:00.215416 11706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:00.212791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57124 (local address 127.9.66.2:44051)
0925 07:30:00.212848 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:00.212850 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:00.212860 (+ 10us) server_negotiation.cc:407] Connection header received
0925 07:30:00.212989 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:00.212990 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:00.213010 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:00.213078 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 07:30:00.213212 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:00.213862 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:00.214629 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:00.214839 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:00.214912 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:00.215093 (+ 181us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 07:30:00.215189 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 07:30:00.215273 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:00.215318 (+ 45us) server_negotiation.cc:299] Negotiation successful
0925 07:30:00.215337 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 07:30:00.215436 11726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:00.212575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.2:44051 (local address 127.0.0.1:57124)
0925 07:30:00.212768 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:00.212781 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 07:30:00.212869 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:30:00.213083 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:30:00.213084 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:30:00.213085 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 07:30:00.213136 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:00.213139 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:00.213947 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:00.213948 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:00.214554 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:00.214556 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:00.214912 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:00.214913 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:00.214976 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:00.214996 (+ 20us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 07:30:00.215253 (+ 257us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 07:30:00.215262 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 07:30:00.215315 (+ 53us) client_negotiation.cc:240] Negotiation successful
0925 07:30:00.215361 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137}
I20260925 07:30:00.217217 11643 raft_consensus.cc:1261] T 35d2ed81f27f4d87bbf02db370e612f1 P f713bc029900475cbefa33049b3bd9a3 [term 1 FOLLOWER]: Refusing update from remote peer d9c405e70885447fb3466fd8ceda0714: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:30:00.217250 11363 raft_consensus.cc:1261] T 35d2ed81f27f4d87bbf02db370e612f1 P a429603a7ba54c9a80d82f953c7114c3 [term 1 FOLLOWER]: Refusing update from remote peer d9c405e70885447fb3466fd8ceda0714: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:30:00.217540 11716 consensus_queue.cc:1035] T 35d2ed81f27f4d87bbf02db370e612f1 P d9c405e70885447fb3466fd8ceda0714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a429603a7ba54c9a80d82f953c7114c3" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 43749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:30:00.217661 11719 consensus_queue.cc:1035] T 35d2ed81f27f4d87bbf02db370e612f1 P d9c405e70885447fb3466fd8ceda0714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f713bc029900475cbefa33049b3bd9a3" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 37053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:30:00.218798 11717 tablet_replica.cc:439] T 35d2ed81f27f4d87bbf02db370e612f1 P a429603a7ba54c9a80d82f953c7114c3: TxnStatusTablet state changed. Reason: New leader d9c405e70885447fb3466fd8ceda0714. Latest consensus state: current_term: 1 leader_uuid: "d9c405e70885447fb3466fd8ceda0714" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a429603a7ba54c9a80d82f953c7114c3" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 43749 } } peers { permanent_uuid: "d9c405e70885447fb3466fd8ceda0714" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 44051 } } peers { permanent_uuid: "f713bc029900475cbefa33049b3bd9a3" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 37053 } } }
I20260925 07:30:00.218899 11717 tablet_replica.cc:442] T 35d2ed81f27f4d87bbf02db370e612f1 P a429603a7ba54c9a80d82f953c7114c3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 07:30:00.218942 11714 tablet_replica.cc:439] T 35d2ed81f27f4d87bbf02db370e612f1 P f713bc029900475cbefa33049b3bd9a3: TxnStatusTablet state changed. Reason: New leader d9c405e70885447fb3466fd8ceda0714. Latest consensus state: current_term: 1 leader_uuid: "d9c405e70885447fb3466fd8ceda0714" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a429603a7ba54c9a80d82f953c7114c3" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 43749 } } peers { permanent_uuid: "d9c405e70885447fb3466fd8ceda0714" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 44051 } } peers { permanent_uuid: "f713bc029900475cbefa33049b3bd9a3" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 37053 } } }
I20260925 07:30:00.219053 11714 tablet_replica.cc:442] T 35d2ed81f27f4d87bbf02db370e612f1 P f713bc029900475cbefa33049b3bd9a3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 07:30:00.220021 11719 tablet_replica.cc:439] T 35d2ed81f27f4d87bbf02db370e612f1 P d9c405e70885447fb3466fd8ceda0714: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d9c405e70885447fb3466fd8ceda0714" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a429603a7ba54c9a80d82f953c7114c3" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 43749 } } peers { permanent_uuid: "d9c405e70885447fb3466fd8ceda0714" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 44051 } } peers { permanent_uuid: "f713bc029900475cbefa33049b3bd9a3" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 37053 } } }
I20260925 07:30:00.220108 11719 tablet_replica.cc:442] T 35d2ed81f27f4d87bbf02db370e612f1 P d9c405e70885447fb3466fd8ceda0714: This TxnStatusTablet replica's current role is: LEADER
I20260925 07:30:00.220175 11734 mvcc.cc:205] Tried to move back new op lower bound from 7333156455287119872 to 7333156455090364416. Current Snapshot: MvccSnapshot[applied={T|T < 7333156455287119872}]
I20260925 07:30:00.220271 11714 tablet_replica.cc:439] T 35d2ed81f27f4d87bbf02db370e612f1 P f713bc029900475cbefa33049b3bd9a3: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d9c405e70885447fb3466fd8ceda0714" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a429603a7ba54c9a80d82f953c7114c3" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 43749 } } peers { permanent_uuid: "d9c405e70885447fb3466fd8ceda0714" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 44051 } } peers { permanent_uuid: "f713bc029900475cbefa33049b3bd9a3" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 37053 } } }
I20260925 07:30:00.220347 11714 tablet_replica.cc:442] T 35d2ed81f27f4d87bbf02db370e612f1 P f713bc029900475cbefa33049b3bd9a3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 07:30:00.220661 11716 tablet_replica.cc:439] T 35d2ed81f27f4d87bbf02db370e612f1 P d9c405e70885447fb3466fd8ceda0714: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d9c405e70885447fb3466fd8ceda0714" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a429603a7ba54c9a80d82f953c7114c3" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 43749 } } peers { permanent_uuid: "d9c405e70885447fb3466fd8ceda0714" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 44051 } } peers { permanent_uuid: "f713bc029900475cbefa33049b3bd9a3" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 37053 } } }
I20260925 07:30:00.220741 11716 tablet_replica.cc:442] T 35d2ed81f27f4d87bbf02db370e612f1 P d9c405e70885447fb3466fd8ceda0714: This TxnStatusTablet replica's current role is: LEADER
I20260925 07:30:00.221087 11717 tablet_replica.cc:439] T 35d2ed81f27f4d87bbf02db370e612f1 P a429603a7ba54c9a80d82f953c7114c3: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d9c405e70885447fb3466fd8ceda0714" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a429603a7ba54c9a80d82f953c7114c3" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 43749 } } peers { permanent_uuid: "d9c405e70885447fb3466fd8ceda0714" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 44051 } } peers { permanent_uuid: "f713bc029900475cbefa33049b3bd9a3" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 37053 } } }
I20260925 07:30:00.221170 11717 tablet_replica.cc:442] T 35d2ed81f27f4d87bbf02db370e612f1 P a429603a7ba54c9a80d82f953c7114c3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 07:30:00.221357 11735 mvcc.cc:205] Tried to move back new op lower bound from 7333156455287119872 to 7333156455090364416. Current Snapshot: MvccSnapshot[applied={T|T < 7333156455287119872}]
I20260925 07:30:00.221700 11736 mvcc.cc:205] Tried to move back new op lower bound from 7333156455287119872 to 7333156455090364416. Current Snapshot: MvccSnapshot[applied={T|T < 7333156455287119872}]
I20260925 07:30:00.224820 11706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:00.222478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57134 (local address 127.9.66.2:44051)
0925 07:30:00.222523 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:00.222525 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:00.222532 (+ 7us) server_negotiation.cc:407] Connection header received
0925 07:30:00.222601 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:00.222602 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:00.222609 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:00.222652 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 07:30:00.222795 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:00.223214 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:00.224240 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:00.224403 (+ 163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:00.224449 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:00.224618 (+ 169us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 07:30:00.224640 (+ 22us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 07:30:00.224675 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:00.224729 (+ 54us) server_negotiation.cc:299] Negotiation successful
0925 07:30:00.224743 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 07:30:00.229852 11706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:00.227279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.2:58421 (local address 127.9.66.2:44051)
0925 07:30:00.227344 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:00.227346 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:00.227354 (+ 8us) server_negotiation.cc:407] Connection header received
0925 07:30:00.227376 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:00.227377 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:00.227384 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:00.227429 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 07:30:00.227575 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:00.227984 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:00.229048 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:00.229236 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:00.229303 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:00.229498 (+ 195us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 07:30:00.229579 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 07:30:00.229662 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:00.229736 (+ 74us) server_negotiation.cc:299] Negotiation successful
0925 07:30:00.229756 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 07:30:00.229907 11555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:00.227064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.2:44051 (local address 127.9.66.2:58421)
0925 07:30:00.227131 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:00.227143 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 07:30:00.227191 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:30:00.227458 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:30:00.227459 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:30:00.227460 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 07:30:00.227506 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:00.227508 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:00.228062 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:00.228063 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:00.228942 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:00.228945 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:00.229322 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:00.229323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:00.229381 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:00.229396 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 07:30:00.229671 (+ 275us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 07:30:00.229673 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 07:30:00.229722 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 07:30:00.229766 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260925 07:30:00.232062 11643 raft_consensus.cc:1261] T 8594719ec3db47189053b42f4e4244c4 P f713bc029900475cbefa33049b3bd9a3 [term 1 FOLLOWER]: Refusing update from remote peer d9c405e70885447fb3466fd8ceda0714: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:30:00.232111 11363 raft_consensus.cc:1261] T 8594719ec3db47189053b42f4e4244c4 P a429603a7ba54c9a80d82f953c7114c3 [term 1 FOLLOWER]: Refusing update from remote peer d9c405e70885447fb3466fd8ceda0714: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:30:00.232229 11719 consensus_queue.cc:1035] T 8594719ec3db47189053b42f4e4244c4 P d9c405e70885447fb3466fd8ceda0714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f713bc029900475cbefa33049b3bd9a3" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 37053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:30:00.232334 11716 consensus_queue.cc:1035] T 8594719ec3db47189053b42f4e4244c4 P d9c405e70885447fb3466fd8ceda0714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a429603a7ba54c9a80d82f953c7114c3" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 43749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:30:00.247534 11203 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46460:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 07:30:00.247650 11203 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46460:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 07:30:00.248391 11203 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c175dfccd77d4fcebebcdc0419cdd1d5: Sending DeleteTablet for 3 replicas of tablet 8594719ec3db47189053b42f4e4244c4
I20260925 07:30:00.248689 11482 tablet_service.cc:1514] Processing DeleteTablet for tablet 8594719ec3db47189053b42f4e4244c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:30:00 UTC) from {username='kudu', principal='kudu/127.9.66.62@KRBTEST.COM'} at 127.0.0.1:57122
I20260925 07:30:00.248778 11343 tablet_service.cc:1514] Processing DeleteTablet for tablet 8594719ec3db47189053b42f4e4244c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:30:00 UTC) from {username='kudu', principal='kudu/127.9.66.62@KRBTEST.COM'} at 127.0.0.1:46822
I20260925 07:30:00.248970 11623 tablet_service.cc:1514] Processing DeleteTablet for tablet 8594719ec3db47189053b42f4e4244c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:30:00 UTC) from {username='kudu', principal='kudu/127.9.66.62@KRBTEST.COM'} at 127.0.0.1:43126
I20260925 07:30:00.249022 11751 tablet_replica.cc:330] T 8594719ec3db47189053b42f4e4244c4 P a429603a7ba54c9a80d82f953c7114c3: stopping tablet replica
I20260925 07:30:00.249119 11751 raft_consensus.cc:2229] T 8594719ec3db47189053b42f4e4244c4 P a429603a7ba54c9a80d82f953c7114c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 07:30:00.249158 11752 tablet_replica.cc:330] T 8594719ec3db47189053b42f4e4244c4 P f713bc029900475cbefa33049b3bd9a3: stopping tablet replica
I20260925 07:30:00.249207 11751 raft_consensus.cc:2258] T 8594719ec3db47189053b42f4e4244c4 P a429603a7ba54c9a80d82f953c7114c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:30:00.249280 11752 raft_consensus.cc:2229] T 8594719ec3db47189053b42f4e4244c4 P f713bc029900475cbefa33049b3bd9a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 07:30:00.249405 11752 raft_consensus.cc:2258] T 8594719ec3db47189053b42f4e4244c4 P f713bc029900475cbefa33049b3bd9a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:30:00.249437 11203 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46460:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 07:30:00.249605 11203 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 07:30:00.249823 11751 ts_tablet_manager.cc:1905] T 8594719ec3db47189053b42f4e4244c4 P a429603a7ba54c9a80d82f953c7114c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 07:30:00.250752 11482 tablet_service.cc:1467] Processing CreateTablet for tablet 5ad7fb1fada640449edee7ff225cb1de (DEFAULT_TABLE table=test-table [id=1d038654c8e74f849c5a7a926d362d00]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:30:00.250735 11343 tablet_service.cc:1467] Processing CreateTablet for tablet 5ad7fb1fada640449edee7ff225cb1de (DEFAULT_TABLE table=test-table [id=1d038654c8e74f849c5a7a926d362d00]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:30:00.250859 11482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ad7fb1fada640449edee7ff225cb1de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:30:00.250893 11343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ad7fb1fada640449edee7ff225cb1de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:30:00.251160 11752 ts_tablet_manager.cc:1905] T 8594719ec3db47189053b42f4e4244c4 P f713bc029900475cbefa33049b3bd9a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 07:30:00.252240 11710 tablet_bootstrap.cc:492] T 5ad7fb1fada640449edee7ff225cb1de P a429603a7ba54c9a80d82f953c7114c3: Bootstrap starting.
I20260925 07:30:00.252240 11712 tablet_bootstrap.cc:492] T 5ad7fb1fada640449edee7ff225cb1de P d9c405e70885447fb3466fd8ceda0714: Bootstrap starting.
I20260925 07:30:00.252635 11712 tablet_bootstrap.cc:654] T 5ad7fb1fada640449edee7ff225cb1de P d9c405e70885447fb3466fd8ceda0714: Neither blocks nor log segments found. Creating new log.
I20260925 07:30:00.252635 11710 tablet_bootstrap.cc:654] T 5ad7fb1fada640449edee7ff225cb1de P a429603a7ba54c9a80d82f953c7114c3: Neither blocks nor log segments found. Creating new log.
I20260925 07:30:00.252772 11752 ts_tablet_manager.cc:1918] T 8594719ec3db47189053b42f4e4244c4 P f713bc029900475cbefa33049b3bd9a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 07:30:00.252836 11752 log.cc:1196] T 8594719ec3db47189053b42f4e4244c4 P f713bc029900475cbefa33049b3bd9a3: Deleting WAL directory at /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790321392278483-9480-0/minicluster-data/ts-2/wal/wals/8594719ec3db47189053b42f4e4244c4
I20260925 07:30:00.253116 11752 ts_tablet_manager.cc:1939] T 8594719ec3db47189053b42f4e4244c4 P f713bc029900475cbefa33049b3bd9a3: Deleting consensus metadata
I20260925 07:30:00.253185 11623 tablet_service.cc:1467] Processing CreateTablet for tablet 5ad7fb1fada640449edee7ff225cb1de (DEFAULT_TABLE table=test-table [id=1d038654c8e74f849c5a7a926d362d00]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:30:00.253290 11623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ad7fb1fada640449edee7ff225cb1de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:30:00.253927 11751 ts_tablet_manager.cc:1918] T 8594719ec3db47189053b42f4e4244c4 P a429603a7ba54c9a80d82f953c7114c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 07:30:00.253986 11751 log.cc:1196] T 8594719ec3db47189053b42f4e4244c4 P a429603a7ba54c9a80d82f953c7114c3: Deleting WAL directory at /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790321392278483-9480-0/minicluster-data/ts-0/wal/wals/8594719ec3db47189053b42f4e4244c4
I20260925 07:30:00.254213 11709 tablet_bootstrap.cc:492] T 5ad7fb1fada640449edee7ff225cb1de P f713bc029900475cbefa33049b3bd9a3: Bootstrap starting.
I20260925 07:30:00.254226 11751 ts_tablet_manager.cc:1939] T 8594719ec3db47189053b42f4e4244c4 P a429603a7ba54c9a80d82f953c7114c3: Deleting consensus metadata
I20260925 07:30:00.254310 11191 catalog_manager.cc:4841] TS f713bc029900475cbefa33049b3bd9a3 (127.9.66.3:37053): tablet 8594719ec3db47189053b42f4e4244c4 (table test-table [id=9653d31c39474dd7abae8ab3bca03624]) successfully deleted
I20260925 07:30:00.254630 11709 tablet_bootstrap.cc:654] T 5ad7fb1fada640449edee7ff225cb1de P f713bc029900475cbefa33049b3bd9a3: Neither blocks nor log segments found. Creating new log.
I20260925 07:30:00.254618 11190 catalog_manager.cc:4841] TS a429603a7ba54c9a80d82f953c7114c3 (127.9.66.1:43749): tablet 8594719ec3db47189053b42f4e4244c4 (table test-table [id=9653d31c39474dd7abae8ab3bca03624]) successfully deleted
I20260925 07:30:00.254794 11750 tablet_replica.cc:330] T 8594719ec3db47189053b42f4e4244c4 P d9c405e70885447fb3466fd8ceda0714: stopping tablet replica
I20260925 07:30:00.254935 11750 raft_consensus.cc:2229] T 8594719ec3db47189053b42f4e4244c4 P d9c405e70885447fb3466fd8ceda0714 [term 1 LEADER]: Raft consensus shutting down.
I20260925 07:30:00.255079 11750 raft_consensus.cc:2258] T 8594719ec3db47189053b42f4e4244c4 P d9c405e70885447fb3466fd8ceda0714 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:30:00.255290 11709 tablet_bootstrap.cc:492] T 5ad7fb1fada640449edee7ff225cb1de P f713bc029900475cbefa33049b3bd9a3: No bootstrap required, opened a new log
I20260925 07:30:00.255336 11709 ts_tablet_manager.cc:1397] T 5ad7fb1fada640449edee7ff225cb1de P f713bc029900475cbefa33049b3bd9a3: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 07:30:00.255501 11750 ts_tablet_manager.cc:1905] T 8594719ec3db47189053b42f4e4244c4 P d9c405e70885447fb3466fd8ceda0714: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 07:30:00.255475 11709 raft_consensus.cc:348] T 5ad7fb1fada640449edee7ff225cb1de P f713bc029900475cbefa33049b3bd9a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a429603a7ba54c9a80d82f953c7114c3" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 43749 } } peers { permanent_uuid: "d9c405e70885447fb3466fd8ceda0714" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 44051 } } peers { permanent_uuid: "f713bc029900475cbefa33049b3bd9a3" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 37053 } }
I20260925 07:30:00.255537 11709 raft_consensus.cc:374] T 5ad7fb1fada640449edee7ff225cb1de P f713bc029900475cbefa33049b3bd9a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:30:00.255556 11709 raft_consensus.cc:729] T 5ad7fb1fada640449edee7ff225cb1de P f713bc029900475cbefa33049b3bd9a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f713bc029900475cbefa33049b3bd9a3, State: Initialized, Role: FOLLOWER
I20260925 07:30:00.255605 11709 consensus_queue.cc:260] T 5ad7fb1fada640449edee7ff225cb1de P f713bc029900475cbefa33049b3bd9a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a429603a7ba54c9a80d82f953c7114c3" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 43749 } } peers { permanent_uuid: "d9c405e70885447fb3466fd8ceda0714" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 44051 } } peers { permanent_uuid: "f713bc029900475cbefa33049b3bd9a3" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 37053 } }
I20260925 07:30:00.255702 11709 ts_tablet_manager.cc:1428] T 5ad7fb1fada640449edee7ff225cb1de P f713bc029900475cbefa33049b3bd9a3: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:00.256786 11714 raft_consensus.cc:482] T 5ad7fb1fada640449edee7ff225cb1de P f713bc029900475cbefa33049b3bd9a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 07:30:00.256847 11714 raft_consensus.cc:504] T 5ad7fb1fada640449edee7ff225cb1de P f713bc029900475cbefa33049b3bd9a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a429603a7ba54c9a80d82f953c7114c3" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 43749 } } peers { permanent_uuid: "d9c405e70885447fb3466fd8ceda0714" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 44051 } } peers { permanent_uuid: "f713bc029900475cbefa33049b3bd9a3" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 37053 } }
I20260925 07:30:00.256906 11750 ts_tablet_manager.cc:1918] T 8594719ec3db47189053b42f4e4244c4 P d9c405e70885447fb3466fd8ceda0714: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 07:30:00.256986 11750 log.cc:1196] T 8594719ec3db47189053b42f4e4244c4 P d9c405e70885447fb3466fd8ceda0714: Deleting WAL directory at /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790321392278483-9480-0/minicluster-data/ts-1/wal/wals/8594719ec3db47189053b42f4e4244c4
I20260925 07:30:00.257117 11714 leader_election.cc:288] T 5ad7fb1fada640449edee7ff225cb1de P f713bc029900475cbefa33049b3bd9a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a429603a7ba54c9a80d82f953c7114c3 (127.9.66.1:43749), d9c405e70885447fb3466fd8ceda0714 (127.9.66.2:44051)
I20260925 07:30:00.257283 11750 ts_tablet_manager.cc:1939] T 8594719ec3db47189053b42f4e4244c4 P d9c405e70885447fb3466fd8ceda0714: Deleting consensus metadata
I20260925 07:30:00.257830 11191 catalog_manager.cc:4841] TS d9c405e70885447fb3466fd8ceda0714 (127.9.66.2:44051): tablet 8594719ec3db47189053b42f4e4244c4 (table test-table [id=9653d31c39474dd7abae8ab3bca03624]) successfully deleted
I20260925 07:30:00.260376 11582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:00.257413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.1:43749 (local address 127.9.66.3:54859)
0925 07:30:00.257492 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:00.257523 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0925 07:30:00.257572 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:30:00.257743 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:30:00.257744 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:30:00.257746 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:30:00.257810 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:00.257812 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:00.258673 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:00.258674 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:00.259379 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:00.259387 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:00.260122 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:00.260123 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:00.260194 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:00.260198 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 07:30:00.260273 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 07:30:00.260309 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260925 07:30:00.260381 11705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:00.257455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.3:54859 (local address 127.9.66.1:43749)
0925 07:30:00.257494 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:00.257496 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:00.257628 (+ 132us) server_negotiation.cc:407] Connection header received
0925 07:30:00.257650 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:00.257651 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:00.257659 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:00.257702 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:30:00.257882 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:00.258584 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:00.259476 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:00.260020 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:00.260101 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:00.260112 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:00.260282 (+ 170us) server_negotiation.cc:299] Negotiation successful
0925 07:30:00.260320 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 07:30:00.260793 11362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad7fb1fada640449edee7ff225cb1de" candidate_uuid: "f713bc029900475cbefa33049b3bd9a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a429603a7ba54c9a80d82f953c7114c3" is_pre_election: true
W20260925 07:30:00.261251 11574 leader_election.cc:341] T 5ad7fb1fada640449edee7ff225cb1de P f713bc029900475cbefa33049b3bd9a3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a429603a7ba54c9a80d82f953c7114c3 (127.9.66.1:43749): Illegal state: must be running to vote when last-logged opid is not known
I20260925 07:30:00.261932 11712 tablet_bootstrap.cc:492] T 5ad7fb1fada640449edee7ff225cb1de P d9c405e70885447fb3466fd8ceda0714: No bootstrap required, opened a new log
I20260925 07:30:00.261986 11712 ts_tablet_manager.cc:1397] T 5ad7fb1fada640449edee7ff225cb1de P d9c405e70885447fb3466fd8ceda0714: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.000s
I20260925 07:30:00.262042 11710 tablet_bootstrap.cc:492] T 5ad7fb1fada640449edee7ff225cb1de P a429603a7ba54c9a80d82f953c7114c3: No bootstrap required, opened a new log
I20260925 07:30:00.262089 11710 ts_tablet_manager.cc:1397] T 5ad7fb1fada640449edee7ff225cb1de P a429603a7ba54c9a80d82f953c7114c3: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.000s
I20260925 07:30:00.262130 11712 raft_consensus.cc:348] T 5ad7fb1fada640449edee7ff225cb1de P d9c405e70885447fb3466fd8ceda0714 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a429603a7ba54c9a80d82f953c7114c3" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 43749 } } peers { permanent_uuid: "d9c405e70885447fb3466fd8ceda0714" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 44051 } } peers { permanent_uuid: "f713bc029900475cbefa33049b3bd9a3" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 37053 } }
I20260925 07:30:00.262185 11712 raft_consensus.cc:374] T 5ad7fb1fada640449edee7ff225cb1de P d9c405e70885447fb3466fd8ceda0714 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:30:00.262203 11712 raft_consensus.cc:729] T 5ad7fb1fada640449edee7ff225cb1de P d9c405e70885447fb3466fd8ceda0714 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9c405e70885447fb3466fd8ceda0714, State: Initialized, Role: FOLLOWER
I20260925 07:30:00.262243 11712 consensus_queue.cc:260] T 5ad7fb1fada640449edee7ff225cb1de P d9c405e70885447fb3466fd8ceda0714 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a429603a7ba54c9a80d82f953c7114c3" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 43749 } } peers { permanent_uuid: "d9c405e70885447fb3466fd8ceda0714" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 44051 } } peers { permanent_uuid: "f713bc029900475cbefa33049b3bd9a3" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 37053 } }
I20260925 07:30:00.262224 11710 raft_consensus.cc:348] T 5ad7fb1fada640449edee7ff225cb1de P a429603a7ba54c9a80d82f953c7114c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a429603a7ba54c9a80d82f953c7114c3" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 43749 } } peers { permanent_uuid: "d9c405e70885447fb3466fd8ceda0714" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 44051 } } peers { permanent_uuid: "f713bc029900475cbefa33049b3bd9a3" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 37053 } }
I20260925 07:30:00.262307 11710 raft_consensus.cc:374] T 5ad7fb1fada640449edee7ff225cb1de P a429603a7ba54c9a80d82f953c7114c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:30:00.262331 11710 raft_consensus.cc:729] T 5ad7fb1fada640449edee7ff225cb1de P a429603a7ba54c9a80d82f953c7114c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a429603a7ba54c9a80d82f953c7114c3, State: Initialized, Role: FOLLOWER
I20260925 07:30:00.262339 11712 ts_tablet_manager.cc:1428] T 5ad7fb1fada640449edee7ff225cb1de P d9c405e70885447fb3466fd8ceda0714: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:00.262373 11710 consensus_queue.cc:260] T 5ad7fb1fada640449edee7ff225cb1de P a429603a7ba54c9a80d82f953c7114c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a429603a7ba54c9a80d82f953c7114c3" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 43749 } } peers { permanent_uuid: "d9c405e70885447fb3466fd8ceda0714" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 44051 } } peers { permanent_uuid: "f713bc029900475cbefa33049b3bd9a3" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 37053 } }
I20260925 07:30:00.262465 11710 ts_tablet_manager.cc:1428] T 5ad7fb1fada640449edee7ff225cb1de P a429603a7ba54c9a80d82f953c7114c3: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 07:30:00.263769 11582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:00.257552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.2:44051 (local address 127.9.66.3:39047)
0925 07:30:00.260489 (+ 2937us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:00.260501 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 07:30:00.260565 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:30:00.260749 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:30:00.260750 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:30:00.260752 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:30:00.260787 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:00.260788 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:00.261650 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:00.261652 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:00.262735 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:00.262737 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:00.263520 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:00.263521 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:00.263580 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:00.263583 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 07:30:00.263663 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 07:30:00.263700 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2925,"thread_start_us":50,"threads_started":1}
I20260925 07:30:00.263772 11706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:00.257671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.3:39047 (local address 127.9.66.2:44051)
0925 07:30:00.257708 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:00.257710 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:00.260544 (+ 2834us) server_negotiation.cc:407] Connection header received
0925 07:30:00.260636 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:00.260637 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:00.260649 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:00.260705 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:30:00.260860 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:00.261574 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:00.262915 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:00.263427 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:00.263510 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:00.263523 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:00.263672 (+ 149us) server_negotiation.cc:299] Negotiation successful
0925 07:30:00.263705 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 07:30:00.263988 11503 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad7fb1fada640449edee7ff225cb1de" candidate_uuid: "f713bc029900475cbefa33049b3bd9a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9c405e70885447fb3466fd8ceda0714" is_pre_election: true
I20260925 07:30:00.264109 11503 raft_consensus.cc:2454] T 5ad7fb1fada640449edee7ff225cb1de P d9c405e70885447fb3466fd8ceda0714 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f713bc029900475cbefa33049b3bd9a3 in term 0.
I20260925 07:30:00.264300 11575 leader_election.cc:302] T 5ad7fb1fada640449edee7ff225cb1de P f713bc029900475cbefa33049b3bd9a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d9c405e70885447fb3466fd8ceda0714, f713bc029900475cbefa33049b3bd9a3; no voters: a429603a7ba54c9a80d82f953c7114c3
I20260925 07:30:00.264473 11714 raft_consensus.cc:2781] T 5ad7fb1fada640449edee7ff225cb1de P f713bc029900475cbefa33049b3bd9a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 07:30:00.264518 11714 raft_consensus.cc:482] T 5ad7fb1fada640449edee7ff225cb1de P f713bc029900475cbefa33049b3bd9a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 07:30:00.264542 11714 raft_consensus.cc:3037] T 5ad7fb1fada640449edee7ff225cb1de P f713bc029900475cbefa33049b3bd9a3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:30:00.265125 11714 raft_consensus.cc:504] T 5ad7fb1fada640449edee7ff225cb1de P f713bc029900475cbefa33049b3bd9a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a429603a7ba54c9a80d82f953c7114c3" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 43749 } } peers { permanent_uuid: "d9c405e70885447fb3466fd8ceda0714" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 44051 } } peers { permanent_uuid: "f713bc029900475cbefa33049b3bd9a3" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 37053 } }
I20260925 07:30:00.265285 11714 leader_election.cc:288] T 5ad7fb1fada640449edee7ff225cb1de P f713bc029900475cbefa33049b3bd9a3 [CANDIDATE]: Term 1 election: Requested vote from peers a429603a7ba54c9a80d82f953c7114c3 (127.9.66.1:43749), d9c405e70885447fb3466fd8ceda0714 (127.9.66.2:44051)
I20260925 07:30:00.265383 11362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad7fb1fada640449edee7ff225cb1de" candidate_uuid: "f713bc029900475cbefa33049b3bd9a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a429603a7ba54c9a80d82f953c7114c3"
I20260925 07:30:00.265452 11362 raft_consensus.cc:3037] T 5ad7fb1fada640449edee7ff225cb1de P a429603a7ba54c9a80d82f953c7114c3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:30:00.265431 11503 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad7fb1fada640449edee7ff225cb1de" candidate_uuid: "f713bc029900475cbefa33049b3bd9a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9c405e70885447fb3466fd8ceda0714"
I20260925 07:30:00.265496 11503 raft_consensus.cc:3037] T 5ad7fb1fada640449edee7ff225cb1de P d9c405e70885447fb3466fd8ceda0714 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:30:00.266095 11362 raft_consensus.cc:2454] T 5ad7fb1fada640449edee7ff225cb1de P a429603a7ba54c9a80d82f953c7114c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f713bc029900475cbefa33049b3bd9a3 in term 1.
I20260925 07:30:00.266256 11574 leader_election.cc:302] T 5ad7fb1fada640449edee7ff225cb1de P f713bc029900475cbefa33049b3bd9a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a429603a7ba54c9a80d82f953c7114c3, f713bc029900475cbefa33049b3bd9a3; no voters:
I20260925 07:30:00.266341 11714 raft_consensus.cc:2781] T 5ad7fb1fada640449edee7ff225cb1de P f713bc029900475cbefa33049b3bd9a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:30:00.266346 11503 raft_consensus.cc:2454] T 5ad7fb1fada640449edee7ff225cb1de P d9c405e70885447fb3466fd8ceda0714 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f713bc029900475cbefa33049b3bd9a3 in term 1.
I20260925 07:30:00.266497 11714 raft_consensus.cc:686] T 5ad7fb1fada640449edee7ff225cb1de P f713bc029900475cbefa33049b3bd9a3 [term 1 LEADER]: Becoming Leader. State: Replica: f713bc029900475cbefa33049b3bd9a3, State: Running, Role: LEADER
I20260925 07:30:00.266570 11714 consensus_queue.cc:237] T 5ad7fb1fada640449edee7ff225cb1de P f713bc029900475cbefa33049b3bd9a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a429603a7ba54c9a80d82f953c7114c3" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 43749 } } peers { permanent_uuid: "d9c405e70885447fb3466fd8ceda0714" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 44051 } } peers { permanent_uuid: "f713bc029900475cbefa33049b3bd9a3" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 37053 } }
I20260925 07:30:00.267184 11203 catalog_manager.cc:5495] T 5ad7fb1fada640449edee7ff225cb1de P f713bc029900475cbefa33049b3bd9a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to f713bc029900475cbefa33049b3bd9a3 (127.9.66.3). New cstate: current_term: 1 leader_uuid: "f713bc029900475cbefa33049b3bd9a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a429603a7ba54c9a80d82f953c7114c3" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 43749 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d9c405e70885447fb3466fd8ceda0714" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 44051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f713bc029900475cbefa33049b3bd9a3" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 37053 } health_report { overall_health: HEALTHY } } }
I20260925 07:30:00.274277 11708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:00.271652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43136 (local address 127.9.66.3:37053)
0925 07:30:00.271688 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:00.271690 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:00.271698 (+ 8us) server_negotiation.cc:407] Connection header received
0925 07:30:00.271766 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:00.271766 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:00.271780 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:00.271837 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 07:30:00.271973 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:00.272398 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:00.273581 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:00.273775 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:00.273830 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:00.274000 (+ 170us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 07:30:00.274087 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 07:30:00.274139 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:00.274207 (+ 68us) server_negotiation.cc:299] Negotiation successful
0925 07:30:00.274226 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 07:30:00.279114 11706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:00.276910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.3:54973 (local address 127.9.66.2:44051)
0925 07:30:00.276950 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:00.276952 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:00.276961 (+ 9us) server_negotiation.cc:407] Connection header received
0925 07:30:00.277020 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:00.277021 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:00.277030 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:00.277094 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 07:30:00.277243 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:00.277746 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:00.278466 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:00.278662 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:00.278730 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:00.278894 (+ 164us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 07:30:00.278922 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 07:30:00.278977 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:00.279035 (+ 58us) server_negotiation.cc:299] Negotiation successful
0925 07:30:00.279050 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 07:30:00.279124 11695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:00.276832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.2:44051 (local address 127.9.66.3:54973)
0925 07:30:00.276899 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:00.276910 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 07:30:00.276947 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:30:00.277097 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:30:00.277098 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:30:00.277099 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 07:30:00.277169 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:00.277170 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:00.277827 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:00.277828 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:00.278374 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:00.278375 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:00.278728 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:00.278729 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:00.278788 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:00.278801 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 07:30:00.278979 (+ 178us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 07:30:00.278981 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 07:30:00.279030 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 07:30:00.279063 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260925 07:30:00.281180 11362 raft_consensus.cc:1261] T 5ad7fb1fada640449edee7ff225cb1de P a429603a7ba54c9a80d82f953c7114c3 [term 1 FOLLOWER]: Refusing update from remote peer f713bc029900475cbefa33049b3bd9a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:30:00.281315 11503 raft_consensus.cc:1261] T 5ad7fb1fada640449edee7ff225cb1de P d9c405e70885447fb3466fd8ceda0714 [term 1 FOLLOWER]: Refusing update from remote peer f713bc029900475cbefa33049b3bd9a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:30:00.281385 11714 consensus_queue.cc:1035] T 5ad7fb1fada640449edee7ff225cb1de P f713bc029900475cbefa33049b3bd9a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a429603a7ba54c9a80d82f953c7114c3" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 43749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:30:00.281548 11714 consensus_queue.cc:1035] T 5ad7fb1fada640449edee7ff225cb1de P f713bc029900475cbefa33049b3bd9a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9c405e70885447fb3466fd8ceda0714" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 44051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:30:00.288931 11555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:00.286174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.3:37053 (local address 127.9.66.2:48859)
0925 07:30:00.286217 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:00.286226 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 07:30:00.286273 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:30:00.286462 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:30:00.286463 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:30:00.286464 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 07:30:00.286498 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:00.286499 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:00.287231 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:00.287232 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:00.288102 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:00.288105 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:00.288591 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:00.288592 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:00.288637 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:00.288641 (+ 4us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 07:30:00.288817 (+ 176us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 07:30:00.288819 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 07:30:00.288852 (+ 33us) client_negotiation.cc:240] Negotiation successful
0925 07:30:00.288886 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260925 07:30:00.288942 11708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:00.286228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.2:48859 (local address 127.9.66.3:37053)
0925 07:30:00.286272 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:00.286274 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:00.286282 (+ 8us) server_negotiation.cc:407] Connection header received
0925 07:30:00.286324 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:00.286325 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:00.286353 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:00.286428 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 07:30:00.286564 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:00.287151 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:00.288221 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:00.288516 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:00.288586 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:00.288710 (+ 124us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 07:30:00.288753 (+ 43us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 07:30:00.288804 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:00.288855 (+ 51us) server_negotiation.cc:299] Negotiation successful
0925 07:30:00.288879 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
W20260925 07:30:00.289235 11736 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260925 07:30:01.813223 11206 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46460:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 07:30:01.813408 11206 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46460:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 07:30:01.814433 11206 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c175dfccd77d4fcebebcdc0419cdd1d5: Sending DeleteTablet for 3 replicas of tablet 5ad7fb1fada640449edee7ff225cb1de
I20260925 07:30:01.814710 11482 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ad7fb1fada640449edee7ff225cb1de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:30:01 UTC) from {username='kudu', principal='kudu/127.9.66.62@KRBTEST.COM'} at 127.0.0.1:57122
I20260925 07:30:01.814726 11343 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ad7fb1fada640449edee7ff225cb1de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:30:01 UTC) from {username='kudu', principal='kudu/127.9.66.62@KRBTEST.COM'} at 127.0.0.1:46822
I20260925 07:30:01.814991 11787 tablet_replica.cc:330] T 5ad7fb1fada640449edee7ff225cb1de P a429603a7ba54c9a80d82f953c7114c3: stopping tablet replica
I20260925 07:30:01.815055 11787 raft_consensus.cc:2229] T 5ad7fb1fada640449edee7ff225cb1de P a429603a7ba54c9a80d82f953c7114c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 07:30:01.815093 11787 raft_consensus.cc:2258] T 5ad7fb1fada640449edee7ff225cb1de P a429603a7ba54c9a80d82f953c7114c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:30:01.815336 11787 ts_tablet_manager.cc:1905] T 5ad7fb1fada640449edee7ff225cb1de P a429603a7ba54c9a80d82f953c7114c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 07:30:01.816150 11623 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ad7fb1fada640449edee7ff225cb1de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:30:01 UTC) from {username='kudu', principal='kudu/127.9.66.62@KRBTEST.COM'} at 127.0.0.1:43126
I20260925 07:30:01.816365 11789 tablet_replica.cc:330] T 5ad7fb1fada640449edee7ff225cb1de P f713bc029900475cbefa33049b3bd9a3: stopping tablet replica
I20260925 07:30:01.816475 11789 raft_consensus.cc:2229] T 5ad7fb1fada640449edee7ff225cb1de P f713bc029900475cbefa33049b3bd9a3 [term 1 LEADER]: Raft consensus shutting down.
I20260925 07:30:01.816635 11789 raft_consensus.cc:2258] T 5ad7fb1fada640449edee7ff225cb1de P f713bc029900475cbefa33049b3bd9a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:30:01.817028 11789 ts_tablet_manager.cc:1905] T 5ad7fb1fada640449edee7ff225cb1de P f713bc029900475cbefa33049b3bd9a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 07:30:01.817135 11787 ts_tablet_manager.cc:1918] T 5ad7fb1fada640449edee7ff225cb1de P a429603a7ba54c9a80d82f953c7114c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 07:30:01.817203 11787 log.cc:1196] T 5ad7fb1fada640449edee7ff225cb1de P a429603a7ba54c9a80d82f953c7114c3: Deleting WAL directory at /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790321392278483-9480-0/minicluster-data/ts-0/wal/wals/5ad7fb1fada640449edee7ff225cb1de
I20260925 07:30:01.817950 11786 tablet_replica.cc:330] T 5ad7fb1fada640449edee7ff225cb1de P d9c405e70885447fb3466fd8ceda0714: stopping tablet replica
I20260925 07:30:01.818028 11786 raft_consensus.cc:2229] T 5ad7fb1fada640449edee7ff225cb1de P d9c405e70885447fb3466fd8ceda0714 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 07:30:01.818107 11786 raft_consensus.cc:2258] T 5ad7fb1fada640449edee7ff225cb1de P d9c405e70885447fb3466fd8ceda0714 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:30:01.818390 11786 ts_tablet_manager.cc:1905] T 5ad7fb1fada640449edee7ff225cb1de P d9c405e70885447fb3466fd8ceda0714: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 07:30:01.818740 11787 ts_tablet_manager.cc:1939] T 5ad7fb1fada640449edee7ff225cb1de P a429603a7ba54c9a80d82f953c7114c3: Deleting consensus metadata
I20260925 07:30:01.818946 11789 ts_tablet_manager.cc:1918] T 5ad7fb1fada640449edee7ff225cb1de P f713bc029900475cbefa33049b3bd9a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 07:30:01.819077 11789 log.cc:1196] T 5ad7fb1fada640449edee7ff225cb1de P f713bc029900475cbefa33049b3bd9a3: Deleting WAL directory at /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790321392278483-9480-0/minicluster-data/ts-2/wal/wals/5ad7fb1fada640449edee7ff225cb1de
I20260925 07:30:01.819090 11190 catalog_manager.cc:4841] TS a429603a7ba54c9a80d82f953c7114c3 (127.9.66.1:43749): tablet 5ad7fb1fada640449edee7ff225cb1de (table test-table [id=1d038654c8e74f849c5a7a926d362d00]) successfully deleted
I20260925 07:30:01.819407 11789 ts_tablet_manager.cc:1939] T 5ad7fb1fada640449edee7ff225cb1de P f713bc029900475cbefa33049b3bd9a3: Deleting consensus metadata
I20260925 07:30:01.819749 11191 catalog_manager.cc:4841] TS f713bc029900475cbefa33049b3bd9a3 (127.9.66.3:37053): tablet 5ad7fb1fada640449edee7ff225cb1de (table test-table [id=1d038654c8e74f849c5a7a926d362d00]) successfully deleted
I20260925 07:30:01.820111 11786 ts_tablet_manager.cc:1918] T 5ad7fb1fada640449edee7ff225cb1de P d9c405e70885447fb3466fd8ceda0714: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 07:30:01.820164 11786 log.cc:1196] T 5ad7fb1fada640449edee7ff225cb1de P d9c405e70885447fb3466fd8ceda0714: Deleting WAL directory at /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790321392278483-9480-0/minicluster-data/ts-1/wal/wals/5ad7fb1fada640449edee7ff225cb1de
I20260925 07:30:01.820400 11786 ts_tablet_manager.cc:1939] T 5ad7fb1fada640449edee7ff225cb1de P d9c405e70885447fb3466fd8ceda0714: Deleting consensus metadata
I20260925 07:30:01.820775 11191 catalog_manager.cc:4841] TS d9c405e70885447fb3466fd8ceda0714 (127.9.66.2:44051): tablet 5ad7fb1fada640449edee7ff225cb1de (table test-table [id=1d038654c8e74f849c5a7a926d362d00]) successfully deleted
Sep 25 07:30:01 dist-test-slave-57fn krb5kdc[11161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321401, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 07:30:01 dist-test-slave-57fn krb5kdc[11161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321401, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.9.66.62@KRBTEST.COM
I20260925 07:30:01.839099 11796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:01.831652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46478 (local address 127.9.66.62:45659)
0925 07:30:01.831807 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:01.831809 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:01.831820 (+ 11us) server_negotiation.cc:407] Connection header received
0925 07:30:01.831906 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:01.831906 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:01.831917 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:01.831994 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:30:01.832567 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:01.833081 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:01.833891 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:01.834092 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:01.834168 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:01.836396 (+ 2228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:30:01.836398 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:30:01.836398 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:30:01.836405 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:30:01.837906 (+ 1501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:01.838272 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:01.838273 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:01.838273 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:01.838331 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:01.838553 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:01.838554 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:01.838554 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:01.838665 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:30:01.838767 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:01.838907 (+ 140us) server_negotiation.cc:299] Negotiation successful
0925 07:30:01.838990 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":68,"threads_started":1}
I20260925 07:30:01.840472 11206 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46478:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 07:30:01.840646 11206 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 07:30:01.842093 11343 tablet_service.cc:1467] Processing CreateTablet for tablet c3b8dc634f0c4e49a4a1802adc37da8e (DEFAULT_TABLE table=test-table [id=10b1557bb3314a8aab01315547cd1dbe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:30:01.842132 11482 tablet_service.cc:1467] Processing CreateTablet for tablet c3b8dc634f0c4e49a4a1802adc37da8e (DEFAULT_TABLE table=test-table [id=10b1557bb3314a8aab01315547cd1dbe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:30:01.842247 11343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3b8dc634f0c4e49a4a1802adc37da8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:30:01.842270 11482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3b8dc634f0c4e49a4a1802adc37da8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:30:01.843330 11798 tablet_bootstrap.cc:492] T c3b8dc634f0c4e49a4a1802adc37da8e P d9c405e70885447fb3466fd8ceda0714: Bootstrap starting.
I20260925 07:30:01.843397 11797 tablet_bootstrap.cc:492] T c3b8dc634f0c4e49a4a1802adc37da8e P a429603a7ba54c9a80d82f953c7114c3: Bootstrap starting.
I20260925 07:30:01.843418 11623 tablet_service.cc:1467] Processing CreateTablet for tablet c3b8dc634f0c4e49a4a1802adc37da8e (DEFAULT_TABLE table=test-table [id=10b1557bb3314a8aab01315547cd1dbe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:30:01.843526 11623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3b8dc634f0c4e49a4a1802adc37da8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:30:01.843811 11797 tablet_bootstrap.cc:654] T c3b8dc634f0c4e49a4a1802adc37da8e P a429603a7ba54c9a80d82f953c7114c3: Neither blocks nor log segments found. Creating new log.
I20260925 07:30:01.844197 11798 tablet_bootstrap.cc:654] T c3b8dc634f0c4e49a4a1802adc37da8e P d9c405e70885447fb3466fd8ceda0714: Neither blocks nor log segments found. Creating new log.
I20260925 07:30:01.844449 11797 tablet_bootstrap.cc:492] T c3b8dc634f0c4e49a4a1802adc37da8e P a429603a7ba54c9a80d82f953c7114c3: No bootstrap required, opened a new log
I20260925 07:30:01.844561 11797 ts_tablet_manager.cc:1397] T c3b8dc634f0c4e49a4a1802adc37da8e P a429603a7ba54c9a80d82f953c7114c3: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 07:30:01.844708 11797 raft_consensus.cc:348] T c3b8dc634f0c4e49a4a1802adc37da8e P a429603a7ba54c9a80d82f953c7114c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a429603a7ba54c9a80d82f953c7114c3" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 43749 } } peers { permanent_uuid: "d9c405e70885447fb3466fd8ceda0714" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 44051 } } peers { permanent_uuid: "f713bc029900475cbefa33049b3bd9a3" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 37053 } }
I20260925 07:30:01.844808 11797 raft_consensus.cc:374] T c3b8dc634f0c4e49a4a1802adc37da8e P a429603a7ba54c9a80d82f953c7114c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:30:01.844868 11797 raft_consensus.cc:729] T c3b8dc634f0c4e49a4a1802adc37da8e P a429603a7ba54c9a80d82f953c7114c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a429603a7ba54c9a80d82f953c7114c3, State: Initialized, Role: FOLLOWER
I20260925 07:30:01.844905 11798 tablet_bootstrap.cc:492] T c3b8dc634f0c4e49a4a1802adc37da8e P d9c405e70885447fb3466fd8ceda0714: No bootstrap required, opened a new log
I20260925 07:30:01.844951 11797 consensus_queue.cc:260] T c3b8dc634f0c4e49a4a1802adc37da8e P a429603a7ba54c9a80d82f953c7114c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a429603a7ba54c9a80d82f953c7114c3" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 43749 } } peers { permanent_uuid: "d9c405e70885447fb3466fd8ceda0714" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 44051 } } peers { permanent_uuid: "f713bc029900475cbefa33049b3bd9a3" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 37053 } }
I20260925 07:30:01.845000 11798 ts_tablet_manager.cc:1397] T c3b8dc634f0c4e49a4a1802adc37da8e P d9c405e70885447fb3466fd8ceda0714: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 07:30:01.845112 11798 raft_consensus.cc:348] T c3b8dc634f0c4e49a4a1802adc37da8e P d9c405e70885447fb3466fd8ceda0714 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a429603a7ba54c9a80d82f953c7114c3" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 43749 } } peers { permanent_uuid: "d9c405e70885447fb3466fd8ceda0714" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 44051 } } peers { permanent_uuid: "f713bc029900475cbefa33049b3bd9a3" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 37053 } }
I20260925 07:30:01.845165 11801 tablet_bootstrap.cc:492] T c3b8dc634f0c4e49a4a1802adc37da8e P f713bc029900475cbefa33049b3bd9a3: Bootstrap starting.
I20260925 07:30:01.845184 11798 raft_consensus.cc:374] T c3b8dc634f0c4e49a4a1802adc37da8e P d9c405e70885447fb3466fd8ceda0714 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:30:01.845208 11798 raft_consensus.cc:729] T c3b8dc634f0c4e49a4a1802adc37da8e P d9c405e70885447fb3466fd8ceda0714 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9c405e70885447fb3466fd8ceda0714, State: Initialized, Role: FOLLOWER
I20260925 07:30:01.845260 11797 ts_tablet_manager.cc:1428] T c3b8dc634f0c4e49a4a1802adc37da8e P a429603a7ba54c9a80d82f953c7114c3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 07:30:01.845269 11798 consensus_queue.cc:260] T c3b8dc634f0c4e49a4a1802adc37da8e P d9c405e70885447fb3466fd8ceda0714 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a429603a7ba54c9a80d82f953c7114c3" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 43749 } } peers { permanent_uuid: "d9c405e70885447fb3466fd8ceda0714" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 44051 } } peers { permanent_uuid: "f713bc029900475cbefa33049b3bd9a3" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 37053 } }
I20260925 07:30:01.845418 11798 ts_tablet_manager.cc:1428] T c3b8dc634f0c4e49a4a1802adc37da8e P d9c405e70885447fb3466fd8ceda0714: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:01.845585 11801 tablet_bootstrap.cc:654] T c3b8dc634f0c4e49a4a1802adc37da8e P f713bc029900475cbefa33049b3bd9a3: Neither blocks nor log segments found. Creating new log.
I20260925 07:30:01.846295 11801 tablet_bootstrap.cc:492] T c3b8dc634f0c4e49a4a1802adc37da8e P f713bc029900475cbefa33049b3bd9a3: No bootstrap required, opened a new log
I20260925 07:30:01.846336 11801 ts_tablet_manager.cc:1397] T c3b8dc634f0c4e49a4a1802adc37da8e P f713bc029900475cbefa33049b3bd9a3: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260925 07:30:01.846450 11801 raft_consensus.cc:348] T c3b8dc634f0c4e49a4a1802adc37da8e P f713bc029900475cbefa33049b3bd9a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a429603a7ba54c9a80d82f953c7114c3" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 43749 } } peers { permanent_uuid: "d9c405e70885447fb3466fd8ceda0714" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 44051 } } peers { permanent_uuid: "f713bc029900475cbefa33049b3bd9a3" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 37053 } }
I20260925 07:30:01.846513 11801 raft_consensus.cc:374] T c3b8dc634f0c4e49a4a1802adc37da8e P f713bc029900475cbefa33049b3bd9a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:30:01.846555 11801 raft_consensus.cc:729] T c3b8dc634f0c4e49a4a1802adc37da8e P f713bc029900475cbefa33049b3bd9a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f713bc029900475cbefa33049b3bd9a3, State: Initialized, Role: FOLLOWER
I20260925 07:30:01.846623 11801 consensus_queue.cc:260] T c3b8dc634f0c4e49a4a1802adc37da8e P f713bc029900475cbefa33049b3bd9a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a429603a7ba54c9a80d82f953c7114c3" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 43749 } } peers { permanent_uuid: "d9c405e70885447fb3466fd8ceda0714" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 44051 } } peers { permanent_uuid: "f713bc029900475cbefa33049b3bd9a3" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 37053 } }
I20260925 07:30:01.846740 11801 ts_tablet_manager.cc:1428] T c3b8dc634f0c4e49a4a1802adc37da8e P f713bc029900475cbefa33049b3bd9a3: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:02.013288 11780 raft_consensus.cc:482] T c3b8dc634f0c4e49a4a1802adc37da8e P f713bc029900475cbefa33049b3bd9a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 07:30:02.013432 11780 raft_consensus.cc:504] T c3b8dc634f0c4e49a4a1802adc37da8e P f713bc029900475cbefa33049b3bd9a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a429603a7ba54c9a80d82f953c7114c3" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 43749 } } peers { permanent_uuid: "d9c405e70885447fb3466fd8ceda0714" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 44051 } } peers { permanent_uuid: "f713bc029900475cbefa33049b3bd9a3" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 37053 } }
I20260925 07:30:02.013653 11780 leader_election.cc:288] T c3b8dc634f0c4e49a4a1802adc37da8e P f713bc029900475cbefa33049b3bd9a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a429603a7ba54c9a80d82f953c7114c3 (127.9.66.1:43749), d9c405e70885447fb3466fd8ceda0714 (127.9.66.2:44051)
I20260925 07:30:02.013937 11362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3b8dc634f0c4e49a4a1802adc37da8e" candidate_uuid: "f713bc029900475cbefa33049b3bd9a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a429603a7ba54c9a80d82f953c7114c3" is_pre_election: true
I20260925 07:30:02.013937 11503 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3b8dc634f0c4e49a4a1802adc37da8e" candidate_uuid: "f713bc029900475cbefa33049b3bd9a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9c405e70885447fb3466fd8ceda0714" is_pre_election: true
I20260925 07:30:02.014046 11362 raft_consensus.cc:2454] T c3b8dc634f0c4e49a4a1802adc37da8e P a429603a7ba54c9a80d82f953c7114c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f713bc029900475cbefa33049b3bd9a3 in term 0.
I20260925 07:30:02.014046 11503 raft_consensus.cc:2454] T c3b8dc634f0c4e49a4a1802adc37da8e P d9c405e70885447fb3466fd8ceda0714 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f713bc029900475cbefa33049b3bd9a3 in term 0.
I20260925 07:30:02.014205 11575 leader_election.cc:302] T c3b8dc634f0c4e49a4a1802adc37da8e P f713bc029900475cbefa33049b3bd9a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d9c405e70885447fb3466fd8ceda0714, f713bc029900475cbefa33049b3bd9a3; no voters:
I20260925 07:30:02.014343 11780 raft_consensus.cc:2781] T c3b8dc634f0c4e49a4a1802adc37da8e P f713bc029900475cbefa33049b3bd9a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 07:30:02.014382 11780 raft_consensus.cc:482] T c3b8dc634f0c4e49a4a1802adc37da8e P f713bc029900475cbefa33049b3bd9a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 07:30:02.014401 11780 raft_consensus.cc:3037] T c3b8dc634f0c4e49a4a1802adc37da8e P f713bc029900475cbefa33049b3bd9a3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:30:02.015132 11780 raft_consensus.cc:504] T c3b8dc634f0c4e49a4a1802adc37da8e P f713bc029900475cbefa33049b3bd9a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a429603a7ba54c9a80d82f953c7114c3" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 43749 } } peers { permanent_uuid: "d9c405e70885447fb3466fd8ceda0714" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 44051 } } peers { permanent_uuid: "f713bc029900475cbefa33049b3bd9a3" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 37053 } }
I20260925 07:30:02.015273 11780 leader_election.cc:288] T c3b8dc634f0c4e49a4a1802adc37da8e P f713bc029900475cbefa33049b3bd9a3 [CANDIDATE]: Term 1 election: Requested vote from peers a429603a7ba54c9a80d82f953c7114c3 (127.9.66.1:43749), d9c405e70885447fb3466fd8ceda0714 (127.9.66.2:44051)
I20260925 07:30:02.015457 11503 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3b8dc634f0c4e49a4a1802adc37da8e" candidate_uuid: "f713bc029900475cbefa33049b3bd9a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9c405e70885447fb3466fd8ceda0714"
I20260925 07:30:02.015551 11503 raft_consensus.cc:3037] T c3b8dc634f0c4e49a4a1802adc37da8e P d9c405e70885447fb3466fd8ceda0714 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:30:02.015475 11362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3b8dc634f0c4e49a4a1802adc37da8e" candidate_uuid: "f713bc029900475cbefa33049b3bd9a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a429603a7ba54c9a80d82f953c7114c3"
I20260925 07:30:02.015610 11362 raft_consensus.cc:3037] T c3b8dc634f0c4e49a4a1802adc37da8e P a429603a7ba54c9a80d82f953c7114c3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:30:02.016207 11503 raft_consensus.cc:2454] T c3b8dc634f0c4e49a4a1802adc37da8e P d9c405e70885447fb3466fd8ceda0714 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f713bc029900475cbefa33049b3bd9a3 in term 1.
I20260925 07:30:02.016215 11362 raft_consensus.cc:2454] T c3b8dc634f0c4e49a4a1802adc37da8e P a429603a7ba54c9a80d82f953c7114c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f713bc029900475cbefa33049b3bd9a3 in term 1.
I20260925 07:30:02.016381 11575 leader_election.cc:302] T c3b8dc634f0c4e49a4a1802adc37da8e P f713bc029900475cbefa33049b3bd9a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d9c405e70885447fb3466fd8ceda0714, f713bc029900475cbefa33049b3bd9a3; no voters:
I20260925 07:30:02.016482 11780 raft_consensus.cc:2781] T c3b8dc634f0c4e49a4a1802adc37da8e P f713bc029900475cbefa33049b3bd9a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:30:02.016551 11780 raft_consensus.cc:686] T c3b8dc634f0c4e49a4a1802adc37da8e P f713bc029900475cbefa33049b3bd9a3 [term 1 LEADER]: Becoming Leader. State: Replica: f713bc029900475cbefa33049b3bd9a3, State: Running, Role: LEADER
I20260925 07:30:02.016609 11780 consensus_queue.cc:237] T c3b8dc634f0c4e49a4a1802adc37da8e P f713bc029900475cbefa33049b3bd9a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a429603a7ba54c9a80d82f953c7114c3" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 43749 } } peers { permanent_uuid: "d9c405e70885447fb3466fd8ceda0714" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 44051 } } peers { permanent_uuid: "f713bc029900475cbefa33049b3bd9a3" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 37053 } }
I20260925 07:30:02.017304 11203 catalog_manager.cc:5495] T c3b8dc634f0c4e49a4a1802adc37da8e P f713bc029900475cbefa33049b3bd9a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to f713bc029900475cbefa33049b3bd9a3 (127.9.66.3). New cstate: current_term: 1 leader_uuid: "f713bc029900475cbefa33049b3bd9a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a429603a7ba54c9a80d82f953c7114c3" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 43749 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d9c405e70885447fb3466fd8ceda0714" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 44051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f713bc029900475cbefa33049b3bd9a3" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 37053 } health_report { overall_health: HEALTHY } } }
I20260925 07:30:02.029685 11806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:02.027297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43148 (local address 127.9.66.3:37053)
0925 07:30:02.027458 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:02.027460 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:02.027466 (+ 6us) server_negotiation.cc:407] Connection header received
0925 07:30:02.027503 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:02.027503 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:02.027513 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:02.027591 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 07:30:02.027703 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:02.028189 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:02.028950 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:02.029158 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:02.029229 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:02.029388 (+ 159us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 07:30:02.029420 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 07:30:02.029489 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:02.029571 (+ 82us) server_negotiation.cc:299] Negotiation successful
0925 07:30:02.029592 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":90,"threads_started":1}
I20260925 07:30:02.031677 11503 raft_consensus.cc:1261] T c3b8dc634f0c4e49a4a1802adc37da8e P d9c405e70885447fb3466fd8ceda0714 [term 1 FOLLOWER]: Refusing update from remote peer f713bc029900475cbefa33049b3bd9a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:30:02.031710 11362 raft_consensus.cc:1261] T c3b8dc634f0c4e49a4a1802adc37da8e P a429603a7ba54c9a80d82f953c7114c3 [term 1 FOLLOWER]: Refusing update from remote peer f713bc029900475cbefa33049b3bd9a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:30:02.031896 11780 consensus_queue.cc:1035] T c3b8dc634f0c4e49a4a1802adc37da8e P f713bc029900475cbefa33049b3bd9a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9c405e70885447fb3466fd8ceda0714" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 44051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:30:02.032145 11780 consensus_queue.cc:1035] T c3b8dc634f0c4e49a4a1802adc37da8e P f713bc029900475cbefa33049b3bd9a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a429603a7ba54c9a80d82f953c7114c3" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 43749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:30:02.041177 11203 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46478:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 07:30:02.041261 11203 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46478:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 07:30:02.041810 11203 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c175dfccd77d4fcebebcdc0419cdd1d5: Sending DeleteTablet for 3 replicas of tablet c3b8dc634f0c4e49a4a1802adc37da8e
I20260925 07:30:02.042086 11482 tablet_service.cc:1514] Processing DeleteTablet for tablet c3b8dc634f0c4e49a4a1802adc37da8e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:30:02 UTC) from {username='kudu', principal='kudu/127.9.66.62@KRBTEST.COM'} at 127.0.0.1:57122
I20260925 07:30:02.042115 11343 tablet_service.cc:1514] Processing DeleteTablet for tablet c3b8dc634f0c4e49a4a1802adc37da8e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:30:02 UTC) from {username='kudu', principal='kudu/127.9.66.62@KRBTEST.COM'} at 127.0.0.1:46822
I20260925 07:30:02.042183 11786 tablet_replica.cc:330] T c3b8dc634f0c4e49a4a1802adc37da8e P d9c405e70885447fb3466fd8ceda0714: stopping tablet replica
I20260925 07:30:02.042243 11786 raft_consensus.cc:2229] T c3b8dc634f0c4e49a4a1802adc37da8e P d9c405e70885447fb3466fd8ceda0714 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 07:30:02.042227 11623 tablet_service.cc:1514] Processing DeleteTablet for tablet c3b8dc634f0c4e49a4a1802adc37da8e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:30:02 UTC) from {username='kudu', principal='kudu/127.9.66.62@KRBTEST.COM'} at 127.0.0.1:43126
I20260925 07:30:02.042279 11786 raft_consensus.cc:2258] T c3b8dc634f0c4e49a4a1802adc37da8e P d9c405e70885447fb3466fd8ceda0714 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:30:02.042308 11789 tablet_replica.cc:330] T c3b8dc634f0c4e49a4a1802adc37da8e P f713bc029900475cbefa33049b3bd9a3: stopping tablet replica
I20260925 07:30:02.042308 11787 tablet_replica.cc:330] T c3b8dc634f0c4e49a4a1802adc37da8e P a429603a7ba54c9a80d82f953c7114c3: stopping tablet replica
I20260925 07:30:02.042353 11787 raft_consensus.cc:2229] T c3b8dc634f0c4e49a4a1802adc37da8e P a429603a7ba54c9a80d82f953c7114c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 07:30:02.042371 11789 raft_consensus.cc:2229] T c3b8dc634f0c4e49a4a1802adc37da8e P f713bc029900475cbefa33049b3bd9a3 [term 1 LEADER]: Raft consensus shutting down.
I20260925 07:30:02.042279 11203 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46478:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 07:30:02.042392 11787 raft_consensus.cc:2258] T c3b8dc634f0c4e49a4a1802adc37da8e P a429603a7ba54c9a80d82f953c7114c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:30:02.042418 11789 raft_consensus.cc:2258] T c3b8dc634f0c4e49a4a1802adc37da8e P f713bc029900475cbefa33049b3bd9a3 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260925 07:30:02.042444 11203 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 07:30:02.042685 11787 ts_tablet_manager.cc:1905] T c3b8dc634f0c4e49a4a1802adc37da8e P a429603a7ba54c9a80d82f953c7114c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 07:30:02.042773 11786 ts_tablet_manager.cc:1905] T c3b8dc634f0c4e49a4a1802adc37da8e P d9c405e70885447fb3466fd8ceda0714: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 07:30:02.042810 11789 ts_tablet_manager.cc:1905] T c3b8dc634f0c4e49a4a1802adc37da8e P f713bc029900475cbefa33049b3bd9a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 07:30:02.044355 11787 ts_tablet_manager.cc:1918] T c3b8dc634f0c4e49a4a1802adc37da8e P a429603a7ba54c9a80d82f953c7114c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 07:30:02.044378 11789 ts_tablet_manager.cc:1918] T c3b8dc634f0c4e49a4a1802adc37da8e P f713bc029900475cbefa33049b3bd9a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 07:30:02.044425 11787 log.cc:1196] T c3b8dc634f0c4e49a4a1802adc37da8e P a429603a7ba54c9a80d82f953c7114c3: Deleting WAL directory at /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790321392278483-9480-0/minicluster-data/ts-0/wal/wals/c3b8dc634f0c4e49a4a1802adc37da8e
I20260925 07:30:02.044435 11789 log.cc:1196] T c3b8dc634f0c4e49a4a1802adc37da8e P f713bc029900475cbefa33049b3bd9a3: Deleting WAL directory at /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790321392278483-9480-0/minicluster-data/ts-2/wal/wals/c3b8dc634f0c4e49a4a1802adc37da8e
I20260925 07:30:02.044406 11786 ts_tablet_manager.cc:1918] T c3b8dc634f0c4e49a4a1802adc37da8e P d9c405e70885447fb3466fd8ceda0714: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 07:30:02.044469 11786 log.cc:1196] T c3b8dc634f0c4e49a4a1802adc37da8e P d9c405e70885447fb3466fd8ceda0714: Deleting WAL directory at /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790321392278483-9480-0/minicluster-data/ts-1/wal/wals/c3b8dc634f0c4e49a4a1802adc37da8e
I20260925 07:30:02.044646 11786 ts_tablet_manager.cc:1939] T c3b8dc634f0c4e49a4a1802adc37da8e P d9c405e70885447fb3466fd8ceda0714: Deleting consensus metadata
I20260925 07:30:02.044651 11787 ts_tablet_manager.cc:1939] T c3b8dc634f0c4e49a4a1802adc37da8e P a429603a7ba54c9a80d82f953c7114c3: Deleting consensus metadata
I20260925 07:30:02.044661 11789 ts_tablet_manager.cc:1939] T c3b8dc634f0c4e49a4a1802adc37da8e P f713bc029900475cbefa33049b3bd9a3: Deleting consensus metadata
I20260925 07:30:02.044952 11191 catalog_manager.cc:4841] TS f713bc029900475cbefa33049b3bd9a3 (127.9.66.3:37053): tablet c3b8dc634f0c4e49a4a1802adc37da8e (table test-table [id=10b1557bb3314a8aab01315547cd1dbe]) successfully deleted
I20260925 07:30:02.045009 11191 catalog_manager.cc:4841] TS d9c405e70885447fb3466fd8ceda0714 (127.9.66.2:44051): tablet c3b8dc634f0c4e49a4a1802adc37da8e (table test-table [id=10b1557bb3314a8aab01315547cd1dbe]) successfully deleted
I20260925 07:30:02.045199 11190 catalog_manager.cc:4841] TS a429603a7ba54c9a80d82f953c7114c3 (127.9.66.1:43749): tablet c3b8dc634f0c4e49a4a1802adc37da8e (table test-table [id=10b1557bb3314a8aab01315547cd1dbe]) successfully deleted
I20260925 07:30:02.045907 11343 tablet_service.cc:1467] Processing CreateTablet for tablet b8f9b2075e86483982ffb204f037374a (DEFAULT_TABLE table=test-table [id=3441a306642942fea98546f2b114cac4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:30:02.045935 11482 tablet_service.cc:1467] Processing CreateTablet for tablet b8f9b2075e86483982ffb204f037374a (DEFAULT_TABLE table=test-table [id=3441a306642942fea98546f2b114cac4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:30:02.045955 11623 tablet_service.cc:1467] Processing CreateTablet for tablet b8f9b2075e86483982ffb204f037374a (DEFAULT_TABLE table=test-table [id=3441a306642942fea98546f2b114cac4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:30:02.046013 11343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8f9b2075e86483982ffb204f037374a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:30:02.046013 11482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8f9b2075e86483982ffb204f037374a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:30:02.046057 11623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8f9b2075e86483982ffb204f037374a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:30:02.046864 11801 tablet_bootstrap.cc:492] T b8f9b2075e86483982ffb204f037374a P f713bc029900475cbefa33049b3bd9a3: Bootstrap starting.
I20260925 07:30:02.047114 11797 tablet_bootstrap.cc:492] T b8f9b2075e86483982ffb204f037374a P a429603a7ba54c9a80d82f953c7114c3: Bootstrap starting.
I20260925 07:30:02.047209 11801 tablet_bootstrap.cc:654] T b8f9b2075e86483982ffb204f037374a P f713bc029900475cbefa33049b3bd9a3: Neither blocks nor log segments found. Creating new log.
I20260925 07:30:02.047720 11797 tablet_bootstrap.cc:654] T b8f9b2075e86483982ffb204f037374a P a429603a7ba54c9a80d82f953c7114c3: Neither blocks nor log segments found. Creating new log.
I20260925 07:30:02.047860 11801 tablet_bootstrap.cc:492] T b8f9b2075e86483982ffb204f037374a P f713bc029900475cbefa33049b3bd9a3: No bootstrap required, opened a new log
I20260925 07:30:02.047914 11801 ts_tablet_manager.cc:1397] T b8f9b2075e86483982ffb204f037374a P f713bc029900475cbefa33049b3bd9a3: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260925 07:30:02.048044 11798 tablet_bootstrap.cc:492] T b8f9b2075e86483982ffb204f037374a P d9c405e70885447fb3466fd8ceda0714: Bootstrap starting.
I20260925 07:30:02.048030 11801 raft_consensus.cc:348] T b8f9b2075e86483982ffb204f037374a P f713bc029900475cbefa33049b3bd9a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a429603a7ba54c9a80d82f953c7114c3" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 43749 } } peers { permanent_uuid: "d9c405e70885447fb3466fd8ceda0714" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 44051 } } peers { permanent_uuid: "f713bc029900475cbefa33049b3bd9a3" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 37053 } }
I20260925 07:30:02.048092 11801 raft_consensus.cc:374] T b8f9b2075e86483982ffb204f037374a P f713bc029900475cbefa33049b3bd9a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:30:02.048112 11801 raft_consensus.cc:729] T b8f9b2075e86483982ffb204f037374a P f713bc029900475cbefa33049b3bd9a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f713bc029900475cbefa33049b3bd9a3, State: Initialized, Role: FOLLOWER
I20260925 07:30:02.048156 11801 consensus_queue.cc:260] T b8f9b2075e86483982ffb204f037374a P f713bc029900475cbefa33049b3bd9a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a429603a7ba54c9a80d82f953c7114c3" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 43749 } } peers { permanent_uuid: "d9c405e70885447fb3466fd8ceda0714" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 44051 } } peers { permanent_uuid: "f713bc029900475cbefa33049b3bd9a3" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 37053 } }
I20260925 07:30:02.048246 11801 ts_tablet_manager.cc:1428] T b8f9b2075e86483982ffb204f037374a P f713bc029900475cbefa33049b3bd9a3: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:02.048326 11798 tablet_bootstrap.cc:654] T b8f9b2075e86483982ffb204f037374a P d9c405e70885447fb3466fd8ceda0714: Neither blocks nor log segments found. Creating new log.
I20260925 07:30:02.048756 11797 tablet_bootstrap.cc:492] T b8f9b2075e86483982ffb204f037374a P a429603a7ba54c9a80d82f953c7114c3: No bootstrap required, opened a new log
I20260925 07:30:02.048807 11797 ts_tablet_manager.cc:1397] T b8f9b2075e86483982ffb204f037374a P a429603a7ba54c9a80d82f953c7114c3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 07:30:02.048904 11798 tablet_bootstrap.cc:492] T b8f9b2075e86483982ffb204f037374a P d9c405e70885447fb3466fd8ceda0714: No bootstrap required, opened a new log
I20260925 07:30:02.048949 11797 raft_consensus.cc:348] T b8f9b2075e86483982ffb204f037374a P a429603a7ba54c9a80d82f953c7114c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a429603a7ba54c9a80d82f953c7114c3" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 43749 } } peers { permanent_uuid: "d9c405e70885447fb3466fd8ceda0714" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 44051 } } peers { permanent_uuid: "f713bc029900475cbefa33049b3bd9a3" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 37053 } }
I20260925 07:30:02.048969 11798 ts_tablet_manager.cc:1397] T b8f9b2075e86483982ffb204f037374a P d9c405e70885447fb3466fd8ceda0714: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 07:30:02.048995 11797 raft_consensus.cc:374] T b8f9b2075e86483982ffb204f037374a P a429603a7ba54c9a80d82f953c7114c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:30:02.049009 11797 raft_consensus.cc:729] T b8f9b2075e86483982ffb204f037374a P a429603a7ba54c9a80d82f953c7114c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a429603a7ba54c9a80d82f953c7114c3, State: Initialized, Role: FOLLOWER
I20260925 07:30:02.049077 11797 consensus_queue.cc:260] T b8f9b2075e86483982ffb204f037374a P a429603a7ba54c9a80d82f953c7114c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a429603a7ba54c9a80d82f953c7114c3" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 43749 } } peers { permanent_uuid: "d9c405e70885447fb3466fd8ceda0714" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 44051 } } peers { permanent_uuid: "f713bc029900475cbefa33049b3bd9a3" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 37053 } }
I20260925 07:30:02.049079 11798 raft_consensus.cc:348] T b8f9b2075e86483982ffb204f037374a P d9c405e70885447fb3466fd8ceda0714 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a429603a7ba54c9a80d82f953c7114c3" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 43749 } } peers { permanent_uuid: "d9c405e70885447fb3466fd8ceda0714" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 44051 } } peers { permanent_uuid: "f713bc029900475cbefa33049b3bd9a3" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 37053 } }
I20260925 07:30:02.049173 11797 ts_tablet_manager.cc:1428] T b8f9b2075e86483982ffb204f037374a P a429603a7ba54c9a80d82f953c7114c3: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:02.049188 11798 raft_consensus.cc:374] T b8f9b2075e86483982ffb204f037374a P d9c405e70885447fb3466fd8ceda0714 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:30:02.049212 11798 raft_consensus.cc:729] T b8f9b2075e86483982ffb204f037374a P d9c405e70885447fb3466fd8ceda0714 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9c405e70885447fb3466fd8ceda0714, State: Initialized, Role: FOLLOWER
I20260925 07:30:02.049250 11798 consensus_queue.cc:260] T b8f9b2075e86483982ffb204f037374a P d9c405e70885447fb3466fd8ceda0714 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a429603a7ba54c9a80d82f953c7114c3" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 43749 } } peers { permanent_uuid: "d9c405e70885447fb3466fd8ceda0714" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 44051 } } peers { permanent_uuid: "f713bc029900475cbefa33049b3bd9a3" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 37053 } }
I20260925 07:30:02.049355 11798 ts_tablet_manager.cc:1428] T b8f9b2075e86483982ffb204f037374a P d9c405e70885447fb3466fd8ceda0714: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 07:30:02.320343 11764 raft_consensus.cc:482] T b8f9b2075e86483982ffb204f037374a P f713bc029900475cbefa33049b3bd9a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 07:30:02.320463 11764 raft_consensus.cc:504] T b8f9b2075e86483982ffb204f037374a P f713bc029900475cbefa33049b3bd9a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a429603a7ba54c9a80d82f953c7114c3" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 43749 } } peers { permanent_uuid: "d9c405e70885447fb3466fd8ceda0714" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 44051 } } peers { permanent_uuid: "f713bc029900475cbefa33049b3bd9a3" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 37053 } }
I20260925 07:30:02.320683 11764 leader_election.cc:288] T b8f9b2075e86483982ffb204f037374a P f713bc029900475cbefa33049b3bd9a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a429603a7ba54c9a80d82f953c7114c3 (127.9.66.1:43749), d9c405e70885447fb3466fd8ceda0714 (127.9.66.2:44051)
I20260925 07:30:02.320960 11363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8f9b2075e86483982ffb204f037374a" candidate_uuid: "f713bc029900475cbefa33049b3bd9a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a429603a7ba54c9a80d82f953c7114c3" is_pre_election: true
I20260925 07:30:02.320986 11503 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8f9b2075e86483982ffb204f037374a" candidate_uuid: "f713bc029900475cbefa33049b3bd9a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9c405e70885447fb3466fd8ceda0714" is_pre_election: true
I20260925 07:30:02.321080 11503 raft_consensus.cc:2454] T b8f9b2075e86483982ffb204f037374a P d9c405e70885447fb3466fd8ceda0714 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f713bc029900475cbefa33049b3bd9a3 in term 0.
I20260925 07:30:02.321084 11363 raft_consensus.cc:2454] T b8f9b2075e86483982ffb204f037374a P a429603a7ba54c9a80d82f953c7114c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f713bc029900475cbefa33049b3bd9a3 in term 0.
I20260925 07:30:02.321290 11574 leader_election.cc:302] T b8f9b2075e86483982ffb204f037374a P f713bc029900475cbefa33049b3bd9a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: a429603a7ba54c9a80d82f953c7114c3, d9c405e70885447fb3466fd8ceda0714, f713bc029900475cbefa33049b3bd9a3; no voters:
I20260925 07:30:02.321436 11764 raft_consensus.cc:2781] T b8f9b2075e86483982ffb204f037374a P f713bc029900475cbefa33049b3bd9a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 07:30:02.321493 11764 raft_consensus.cc:482] T b8f9b2075e86483982ffb204f037374a P f713bc029900475cbefa33049b3bd9a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 07:30:02.321580 11764 raft_consensus.cc:3037] T b8f9b2075e86483982ffb204f037374a P f713bc029900475cbefa33049b3bd9a3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:30:02.322309 11764 raft_consensus.cc:504] T b8f9b2075e86483982ffb204f037374a P f713bc029900475cbefa33049b3bd9a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a429603a7ba54c9a80d82f953c7114c3" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 43749 } } peers { permanent_uuid: "d9c405e70885447fb3466fd8ceda0714" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 44051 } } peers { permanent_uuid: "f713bc029900475cbefa33049b3bd9a3" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 37053 } }
I20260925 07:30:02.322466 11764 leader_election.cc:288] T b8f9b2075e86483982ffb204f037374a P f713bc029900475cbefa33049b3bd9a3 [CANDIDATE]: Term 1 election: Requested vote from peers a429603a7ba54c9a80d82f953c7114c3 (127.9.66.1:43749), d9c405e70885447fb3466fd8ceda0714 (127.9.66.2:44051)
I20260925 07:30:02.322618 11503 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8f9b2075e86483982ffb204f037374a" candidate_uuid: "f713bc029900475cbefa33049b3bd9a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9c405e70885447fb3466fd8ceda0714"
I20260925 07:30:02.322643 11363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8f9b2075e86483982ffb204f037374a" candidate_uuid: "f713bc029900475cbefa33049b3bd9a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a429603a7ba54c9a80d82f953c7114c3"
I20260925 07:30:02.322713 11363 raft_consensus.cc:3037] T b8f9b2075e86483982ffb204f037374a P a429603a7ba54c9a80d82f953c7114c3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:30:02.322713 11503 raft_consensus.cc:3037] T b8f9b2075e86483982ffb204f037374a P d9c405e70885447fb3466fd8ceda0714 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:30:02.323491 11363 raft_consensus.cc:2454] T b8f9b2075e86483982ffb204f037374a P a429603a7ba54c9a80d82f953c7114c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f713bc029900475cbefa33049b3bd9a3 in term 1.
I20260925 07:30:02.323490 11503 raft_consensus.cc:2454] T b8f9b2075e86483982ffb204f037374a P d9c405e70885447fb3466fd8ceda0714 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f713bc029900475cbefa33049b3bd9a3 in term 1.
I20260925 07:30:02.323673 11575 leader_election.cc:302] T b8f9b2075e86483982ffb204f037374a P f713bc029900475cbefa33049b3bd9a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d9c405e70885447fb3466fd8ceda0714, f713bc029900475cbefa33049b3bd9a3; no voters:
I20260925 07:30:02.323763 11764 raft_consensus.cc:2781] T b8f9b2075e86483982ffb204f037374a P f713bc029900475cbefa33049b3bd9a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:30:02.323804 11764 raft_consensus.cc:686] T b8f9b2075e86483982ffb204f037374a P f713bc029900475cbefa33049b3bd9a3 [term 1 LEADER]: Becoming Leader. State: Replica: f713bc029900475cbefa33049b3bd9a3, State: Running, Role: LEADER
I20260925 07:30:02.323853 11764 consensus_queue.cc:237] T b8f9b2075e86483982ffb204f037374a P f713bc029900475cbefa33049b3bd9a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a429603a7ba54c9a80d82f953c7114c3" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 43749 } } peers { permanent_uuid: "d9c405e70885447fb3466fd8ceda0714" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 44051 } } peers { permanent_uuid: "f713bc029900475cbefa33049b3bd9a3" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 37053 } }
I20260925 07:30:02.324378 11206 catalog_manager.cc:5495] T b8f9b2075e86483982ffb204f037374a P f713bc029900475cbefa33049b3bd9a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to f713bc029900475cbefa33049b3bd9a3 (127.9.66.3). New cstate: current_term: 1 leader_uuid: "f713bc029900475cbefa33049b3bd9a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a429603a7ba54c9a80d82f953c7114c3" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 43749 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d9c405e70885447fb3466fd8ceda0714" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 44051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f713bc029900475cbefa33049b3bd9a3" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 37053 } health_report { overall_health: HEALTHY } } }
I20260925 07:30:02.332590 11503 raft_consensus.cc:1261] T b8f9b2075e86483982ffb204f037374a P d9c405e70885447fb3466fd8ceda0714 [term 1 FOLLOWER]: Refusing update from remote peer f713bc029900475cbefa33049b3bd9a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:30:02.332590 11363 raft_consensus.cc:1261] T b8f9b2075e86483982ffb204f037374a P a429603a7ba54c9a80d82f953c7114c3 [term 1 FOLLOWER]: Refusing update from remote peer f713bc029900475cbefa33049b3bd9a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:30:02.332813 11764 consensus_queue.cc:1035] T b8f9b2075e86483982ffb204f037374a P f713bc029900475cbefa33049b3bd9a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9c405e70885447fb3466fd8ceda0714" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 44051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:30:02.332902 11777 consensus_queue.cc:1035] T b8f9b2075e86483982ffb204f037374a P f713bc029900475cbefa33049b3bd9a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a429603a7ba54c9a80d82f953c7114c3" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 43749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 07:30:02.337204 11768 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260925 07:30:03.854425 11203 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46478:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 07:30:03.854513 11203 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46478:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 07:30:03.855283 11203 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c175dfccd77d4fcebebcdc0419cdd1d5: Sending DeleteTablet for 3 replicas of tablet b8f9b2075e86483982ffb204f037374a
I20260925 07:30:03.855554 11343 tablet_service.cc:1514] Processing DeleteTablet for tablet b8f9b2075e86483982ffb204f037374a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:30:03 UTC) from {username='kudu', principal='kudu/127.9.66.62@KRBTEST.COM'} at 127.0.0.1:46822
I20260925 07:30:03.855651 11482 tablet_service.cc:1514] Processing DeleteTablet for tablet b8f9b2075e86483982ffb204f037374a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:30:03 UTC) from {username='kudu', principal='kudu/127.9.66.62@KRBTEST.COM'} at 127.0.0.1:57122
I20260925 07:30:03.855696 11623 tablet_service.cc:1514] Processing DeleteTablet for tablet b8f9b2075e86483982ffb204f037374a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:30:03 UTC) from {username='kudu', principal='kudu/127.9.66.62@KRBTEST.COM'} at 127.0.0.1:43126
I20260925 07:30:03.855753 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 11278
I20260925 07:30:03.856519 11840 tablet_replica.cc:330] T b8f9b2075e86483982ffb204f037374a P f713bc029900475cbefa33049b3bd9a3: stopping tablet replica
I20260925 07:30:03.856590 11840 raft_consensus.cc:2229] T b8f9b2075e86483982ffb204f037374a P f713bc029900475cbefa33049b3bd9a3 [term 1 LEADER]: Raft consensus shutting down.
I20260925 07:30:03.856642 11840 raft_consensus.cc:2258] T b8f9b2075e86483982ffb204f037374a P f713bc029900475cbefa33049b3bd9a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:30:03.856730 11841 tablet_replica.cc:330] T b8f9b2075e86483982ffb204f037374a P d9c405e70885447fb3466fd8ceda0714: stopping tablet replica
I20260925 07:30:03.856779 11841 raft_consensus.cc:2229] T b8f9b2075e86483982ffb204f037374a P d9c405e70885447fb3466fd8ceda0714 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 07:30:03.856812 11841 raft_consensus.cc:2258] T b8f9b2075e86483982ffb204f037374a P d9c405e70885447fb3466fd8ceda0714 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:30:03.857039 11840 ts_tablet_manager.cc:1905] T b8f9b2075e86483982ffb204f037374a P f713bc029900475cbefa33049b3bd9a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 07:30:03.858999 11841 ts_tablet_manager.cc:1905] T b8f9b2075e86483982ffb204f037374a P d9c405e70885447fb3466fd8ceda0714: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 07:30:03.859090 11840 ts_tablet_manager.cc:1918] T b8f9b2075e86483982ffb204f037374a P f713bc029900475cbefa33049b3bd9a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 07:30:03.859185 11840 log.cc:1196] T b8f9b2075e86483982ffb204f037374a P f713bc029900475cbefa33049b3bd9a3: Deleting WAL directory at /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790321392278483-9480-0/minicluster-data/ts-2/wal/wals/b8f9b2075e86483982ffb204f037374a
I20260925 07:30:03.859462 11840 ts_tablet_manager.cc:1939] T b8f9b2075e86483982ffb204f037374a P f713bc029900475cbefa33049b3bd9a3: Deleting consensus metadata
I20260925 07:30:03.859804 11191 catalog_manager.cc:4841] TS f713bc029900475cbefa33049b3bd9a3 (127.9.66.3:37053): tablet b8f9b2075e86483982ffb204f037374a (table test-table [id=3441a306642942fea98546f2b114cac4]) successfully deleted
I20260925 07:30:03.860842 11841 ts_tablet_manager.cc:1918] T b8f9b2075e86483982ffb204f037374a P d9c405e70885447fb3466fd8ceda0714: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 07:30:03.860911 11841 log.cc:1196] T b8f9b2075e86483982ffb204f037374a P d9c405e70885447fb3466fd8ceda0714: Deleting WAL directory at /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790321392278483-9480-0/minicluster-data/ts-1/wal/wals/b8f9b2075e86483982ffb204f037374a
I20260925 07:30:03.861121 11841 ts_tablet_manager.cc:1939] T b8f9b2075e86483982ffb204f037374a P d9c405e70885447fb3466fd8ceda0714: Deleting consensus metadata
I20260925 07:30:03.861675 11191 catalog_manager.cc:4841] TS d9c405e70885447fb3466fd8ceda0714 (127.9.66.2:44051): tablet b8f9b2075e86483982ffb204f037374a (table test-table [id=3441a306642942fea98546f2b114cac4]) successfully deleted
W20260925 07:30:03.862596 11190 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.66.1:43749 (error 108)
I20260925 07:30:03.862865 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 11418
I20260925 07:30:03.865679 11843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:03.862994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.1:43749 (local address 127.0.0.1:37446)
0925 07:30:03.865600 (+ 2606us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:03.865649 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.1:43749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2579,"thread_start_us":59,"threads_started":1}
W20260925 07:30:03.865787 11190 catalog_manager.cc:4584] TS a429603a7ba54c9a80d82f953c7114c3 (127.9.66.1:43749): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b8f9b2075e86483982ffb204f037374a: Network error: Client connection negotiation failed: client connection to 127.9.66.1:43749: connect: Connection refused (error 111)
I20260925 07:30:03.868826 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 11558
I20260925 07:30:03.874050 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 11174
2026-09-25T07:30:03Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5741 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790321392278483-9480-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 07:30:03 dist-test-slave-57fn krb5kdc[11846](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 07:30:03 dist-test-slave-57fn krb5kdc[11846](info): set up 2 sockets
Sep 25 07:30:03 dist-test-slave-57fn krb5kdc[11846](info): commencing operation
krb5kdc: starting...
W20260925 07:30:04.901202 9480 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790321392278483-9480-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790321392278483-9480-0/krb5kdc/test-user.keytab.
Sep 25 07:30:04 dist-test-slave-57fn krb5kdc[11846](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321404, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T07:30:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T07:30:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.66.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.66.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.66.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab.
Entry for principal kudu/127.9.66.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab.
I20260925 07:30:05.014626 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790321392278483-9480-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790321392278483-9480-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790321392278483-9480-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.66.62:37957
--webserver_interface=127.9.66.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab
--principal=kudu/127.9.66.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:46883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.66.62:37957
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:30:05.091280 11859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:30:05.091478 11859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:30:05.091500 11859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:30:05.092643 11859 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:30:05.092681 11859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:30:05.092695 11859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:30:05.092707 11859 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:30:05.092720 11859 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:30:05.094139 11859 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:46883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790321392278483-9480-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790321392278483-9480-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.66.62:37957
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab
--principal=kudu/127.9.66.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.66.62:37957
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.66.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790321392278483-9480-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:30:05.094367 11859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:30:05.094596 11859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:30:05.099664 11867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:30:05.099671 11859 server_base.cc:1034] running on GCE node
W20260925 07:30:05.099795 11865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:30:05.099732 11864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:30:05.100133 11859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:30:05.100445 11859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:30:05.101570 11859 hybrid_clock.cc:648] HybridClock initialized: now 1790321405101555 us; error 28 us; skew 500 ppm
Sep 25 07:30:05 dist-test-slave-57fn krb5kdc[11846](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321405, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:30:05.108539 11859 init.cc:377] Logged in from keytab as kudu/127.9.66.62@KRBTEST.COM (short username kudu)
I20260925 07:30:05.109030 11859 webserver.cc:460] Webserver started at http://127.9.66.62:35533/ using document root <none> and password file <none>
I20260925 07:30:05.109253 11859 fs_manager.cc:359] Metadata directory not provided
I20260925 07:30:05.109330 11859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:30:05.109479 11859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:30:05.110566 11859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790321392278483-9480-0/minicluster-data/master-0/data/instance:
uuid: "8251314706d34165b76e4857cac03f89"
format_stamp: "Formatted at 2026-09-25 07:30:05 on dist-test-slave-57fn"
server_key: "137215d19604da8d8dd646c85fe94d29"
server_key_iv: "d9dce94700eb9b4a528a03ccb2d69229"
server_key_version: "encryptionkey@0"
I20260925 07:30:05.110947 11859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790321392278483-9480-0/minicluster-data/master-0/wal/instance:
uuid: "8251314706d34165b76e4857cac03f89"
format_stamp: "Formatted at 2026-09-25 07:30:05 on dist-test-slave-57fn"
server_key: "137215d19604da8d8dd646c85fe94d29"
server_key_iv: "d9dce94700eb9b4a528a03ccb2d69229"
server_key_version: "encryptionkey@0"
I20260925 07:30:05.112210 11859 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 07:30:05.112884 11874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:05.113062 11859 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 07:30:05.113145 11859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790321392278483-9480-0/minicluster-data/master-0/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790321392278483-9480-0/minicluster-data/master-0/wal
uuid: "8251314706d34165b76e4857cac03f89"
format_stamp: "Formatted at 2026-09-25 07:30:05 on dist-test-slave-57fn"
server_key: "137215d19604da8d8dd646c85fe94d29"
server_key_iv: "d9dce94700eb9b4a528a03ccb2d69229"
server_key_version: "encryptionkey@0"
I20260925 07:30:05.113227 11859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790321392278483-9480-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790321392278483-9480-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790321392278483-9480-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:30:05.130703 11859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:30:05.132488 11859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:30:05.132647 11859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:30:05.136561 11859 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.62:37957
I20260925 07:30:05.136608 11926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.62:37957 every 8 connection(s)
I20260925 07:30:05.136973 11859 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
I20260925 07:30:05.137835 11927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:30:05.139132 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 11859
I20260925 07:30:05.139226 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790321392278483-9480-0/minicluster-data/master-0/wal/instance
I20260925 07:30:05.139407 9480 external_mini_cluster.cc:1428] Setting key 39583ffbbc2ef0a7a7fc6ce275c36703
I20260925 07:30:05.140409 11927 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8251314706d34165b76e4857cac03f89: Bootstrap starting.
I20260925 07:30:05.141110 11927 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8251314706d34165b76e4857cac03f89: Neither blocks nor log segments found. Creating new log.
I20260925 07:30:05.141395 11927 log.cc:824] T 00000000000000000000000000000000 P 8251314706d34165b76e4857cac03f89: Log is configured to *not* fsync() on all Append() calls
I20260925 07:30:05.142047 11927 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8251314706d34165b76e4857cac03f89: No bootstrap required, opened a new log
I20260925 07:30:05.143755 11927 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8251314706d34165b76e4857cac03f89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8251314706d34165b76e4857cac03f89" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 37957 } }
I20260925 07:30:05.143913 11927 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8251314706d34165b76e4857cac03f89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:30:05.143975 11927 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8251314706d34165b76e4857cac03f89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8251314706d34165b76e4857cac03f89, State: Initialized, Role: FOLLOWER
I20260925 07:30:05.144384 11927 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8251314706d34165b76e4857cac03f89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8251314706d34165b76e4857cac03f89" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 37957 } }
I20260925 07:30:05.144490 11927 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8251314706d34165b76e4857cac03f89 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 07:30:05.144536 11927 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8251314706d34165b76e4857cac03f89 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 07:30:05.144608 11927 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8251314706d34165b76e4857cac03f89 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:30:05.145136 11927 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8251314706d34165b76e4857cac03f89 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8251314706d34165b76e4857cac03f89" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 37957 } }
I20260925 07:30:05.145246 11927 leader_election.cc:302] T 00000000000000000000000000000000 P 8251314706d34165b76e4857cac03f89 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8251314706d34165b76e4857cac03f89; no voters:
I20260925 07:30:05.145443 11927 leader_election.cc:288] T 00000000000000000000000000000000 P 8251314706d34165b76e4857cac03f89 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 07:30:05.145558 11932 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8251314706d34165b76e4857cac03f89 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:30:05.145745 11932 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8251314706d34165b76e4857cac03f89 [term 1 LEADER]: Becoming Leader. State: Replica: 8251314706d34165b76e4857cac03f89, State: Running, Role: LEADER
I20260925 07:30:05.145730 11927 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8251314706d34165b76e4857cac03f89 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:30:05.145891 11932 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8251314706d34165b76e4857cac03f89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8251314706d34165b76e4857cac03f89" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 37957 } }
Sep 25 07:30:05 dist-test-slave-57fn krb5kdc[11846](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321404, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.66.62@KRBTEST.COM
I20260925 07:30:05.146926 11934 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8251314706d34165b76e4857cac03f89 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8251314706d34165b76e4857cac03f89. Latest consensus state: current_term: 1 leader_uuid: "8251314706d34165b76e4857cac03f89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8251314706d34165b76e4857cac03f89" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 37957 } } }
I20260925 07:30:05.147094 11934 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8251314706d34165b76e4857cac03f89 [sys.catalog]: This master's current role is: LEADER
I20260925 07:30:05.147253 11933 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8251314706d34165b76e4857cac03f89 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8251314706d34165b76e4857cac03f89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8251314706d34165b76e4857cac03f89" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 37957 } } }
I20260925 07:30:05.147312 11933 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8251314706d34165b76e4857cac03f89 [sys.catalog]: This master's current role is: LEADER
W20260925 07:30:05.147912 11947 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8251314706d34165b76e4857cac03f89: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 07:30:05.147964 11947 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 07:30:05.148471 11948 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 07:30:05.148655 11948 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 07:30:05.150110 11930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:05.139909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59066 (local address 127.9.66.62:37957)
0925 07:30:05.140088 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:05.140097 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:05.140128 (+ 31us) server_negotiation.cc:407] Connection header received
0925 07:30:05.140644 (+ 516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:05.140652 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:05.140715 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:05.140871 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:30:05.141669 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:05.142640 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:05.143422 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:05.143819 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:05.143935 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:05.146538 (+ 2603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:30:05.146542 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:30:05.146542 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:30:05.146553 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:30:05.148394 (+ 1841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:05.148811 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:05.148813 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:05.148814 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:05.148908 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:05.149134 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:05.149135 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:05.149135 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:05.149407 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:30:05.149551 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:05.149798 (+ 247us) server_negotiation.cc:299] Negotiation successful
0925 07:30:05.149926 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":80,"threads_started":1}
I20260925 07:30:05.150242 11948 catalog_manager.cc:1347] Generated new cluster ID: fd96b1419a2f49ffb78c94bc38053f1b
I20260925 07:30:05.150272 11948 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 07:30:05.189882 11948 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 07:30:05.190454 11948 catalog_manager.cc:1504] Loading token signing keys...
I20260925 07:30:05.196602 11948 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8251314706d34165b76e4857cac03f89: Generated new TSK 0
I20260925 07:30:05.196791 11948 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.66.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.66.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.66.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.1.keytab.
Entry for principal kudu/127.9.66.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.1.keytab.
I20260925 07:30:05.224157 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790321392278483-9480-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790321392278483-9480-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790321392278483-9480-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790321392278483-9480-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.66.1:0
--local_ip_for_outbound_sockets=127.9.66.1
--webserver_interface=127.9.66.1
--webserver_port=0
--tserver_master_addrs=127.9.66.62:37957
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.1.keytab
--principal=kudu/127.9.66.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:46883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:30:05.303726 11953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:30:05.303911 11953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:30:05.303933 11953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:30:05.305049 11953 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:30:05.305086 11953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:30:05.305111 11953 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.66.1
I20260925 07:30:05.306569 11953 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:46883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790321392278483-9480-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790321392278483-9480-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.1.keytab
--principal=kudu/127.9.66.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.66.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790321392278483-9480-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.66.1
--webserver_port=0
--tserver_master_addrs=127.9.66.62:37957
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11953
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.66.1
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790321392278483-9480-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:30:05.306790 11953 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:30:05.307039 11953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 07:30:05.311980 11953 server_base.cc:1034] running on GCE node
W20260925 07:30:05.311939 11958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:30:05.312043 11961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:30:05.311949 11959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:30:05.312373 11953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:30:05.312631 11953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:30:05.313786 11953 hybrid_clock.cc:648] HybridClock initialized: now 1790321405313771 us; error 48 us; skew 500 ppm
Sep 25 07:30:05 dist-test-slave-57fn krb5kdc[11846](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321405, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:30:05.316188 11953 init.cc:377] Logged in from keytab as kudu/127.9.66.1@KRBTEST.COM (short username kudu)
I20260925 07:30:05.316665 11953 webserver.cc:460] Webserver started at http://127.9.66.1:45693/ using document root <none> and password file <none>
I20260925 07:30:05.316938 11953 fs_manager.cc:359] Metadata directory not provided
I20260925 07:30:05.317003 11953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:30:05.317143 11953 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:30:05.318159 11953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790321392278483-9480-0/minicluster-data/ts-0/data/instance:
uuid: "08bfd4496c8c4c7da36aa0288bac28a4"
format_stamp: "Formatted at 2026-09-25 07:30:05 on dist-test-slave-57fn"
server_key: "712363a403da291882f9c1a0f4fde320"
server_key_iv: "75a2fd0590469e03b2975c3941ed78f5"
server_key_version: "encryptionkey@0"
I20260925 07:30:05.318495 11953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790321392278483-9480-0/minicluster-data/ts-0/wal/instance:
uuid: "08bfd4496c8c4c7da36aa0288bac28a4"
format_stamp: "Formatted at 2026-09-25 07:30:05 on dist-test-slave-57fn"
server_key: "712363a403da291882f9c1a0f4fde320"
server_key_iv: "75a2fd0590469e03b2975c3941ed78f5"
server_key_version: "encryptionkey@0"
I20260925 07:30:05.319691 11953 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 07:30:05.320399 11968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:05.320559 11953 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 07:30:05.320663 11953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790321392278483-9480-0/minicluster-data/ts-0/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790321392278483-9480-0/minicluster-data/ts-0/wal
uuid: "08bfd4496c8c4c7da36aa0288bac28a4"
format_stamp: "Formatted at 2026-09-25 07:30:05 on dist-test-slave-57fn"
server_key: "712363a403da291882f9c1a0f4fde320"
server_key_iv: "75a2fd0590469e03b2975c3941ed78f5"
server_key_version: "encryptionkey@0"
I20260925 07:30:05.320744 11953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790321392278483-9480-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790321392278483-9480-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790321392278483-9480-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:30:05.330380 11953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:30:05.332026 11953 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:30:05.332171 11953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:30:05.332367 11953 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:30:05.332687 11953 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:30:05.332739 11953 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:05.332777 11953 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:30:05.332804 11953 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:05.338387 11953 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.1:43643
I20260925 07:30:05.338433 12081 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.1:43643 every 8 connection(s)
I20260925 07:30:05.338701 11953 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790321392278483-9480-0/minicluster-data/ts-0/data/info.pb
Sep 25 07:30:05 dist-test-slave-57fn krb5kdc[11846](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321405, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.1@KRBTEST.COM for kudu/127.9.66.62@KRBTEST.COM
I20260925 07:30:05.347579 11930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:05.339408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.1:51881 (local address 127.9.66.62:37957)
0925 07:30:05.339485 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:05.339487 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:05.339691 (+ 204us) server_negotiation.cc:407] Connection header received
0925 07:30:05.340141 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:05.340142 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:05.340152 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:05.340199 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:30:05.341063 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:05.341622 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:05.342574 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:05.342757 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:05.342805 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:05.344818 (+ 2013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:30:05.344819 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:30:05.344819 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:30:05.344824 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:30:05.346245 (+ 1421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:05.346665 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:05.346666 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:05.346666 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:05.346716 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:05.347023 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:05.347023 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:05.347024 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:05.347120 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:30:05.347168 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:05.347386 (+ 218us) server_negotiation.cc:299] Negotiation successful
0925 07:30:05.347517 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 07:30:05.347708 12084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:05.339378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:37957 (local address 127.9.66.1:51881)
0925 07:30:05.339575 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:05.339608 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 07:30:05.340066 (+ 458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:30:05.340248 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:30:05.340251 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:30:05.340808 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:30:05.340989 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:05.340993 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:05.341697 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:05.341697 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:05.342503 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:05.342505 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:05.342830 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:05.342831 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:05.342921 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:05.343440 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:30:05.343448 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:30:05.344711 (+ 1263us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:30:05.346313 (+ 1602us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:30:05.346319 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:30:05.346327 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:30:05.346593 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:30:05.346777 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:30:05.346777 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:30:05.346777 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:30:05.346969 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:30:05.347173 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:30:05.347177 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:30:05.347280 (+ 103us) client_negotiation.cc:769] Sending connection context
0925 07:30:05.347363 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 07:30:05.347529 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":118,"threads_started":1}
I20260925 07:30:05.348171 12082 heartbeater.cc:347] Connected to a master server at 127.9.66.62:37957
I20260925 07:30:05.348258 12082 heartbeater.cc:464] Registering TS with master...
I20260925 07:30:05.348388 12082 heartbeater.cc:511] Master 127.9.66.62:37957 requested a full tablet report, sending...
I20260925 07:30:05.348860 11891 ts_manager.cc:194] Registered new tserver with Master: 08bfd4496c8c4c7da36aa0288bac28a4 (127.9.66.1:43643)
I20260925 07:30:05.348870 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 11953
I20260925 07:30:05.348973 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790321392278483-9480-0/minicluster-data/ts-0/wal/instance
I20260925 07:30:05.349153 9480 external_mini_cluster.cc:1428] Setting key 5b09498e29f00332a8d3eb8aded7c90a
I20260925 07:30:05.349666 11891 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.66.1@KRBTEST.COM'} at 127.9.66.1:51881
WARNING: no policy specified for kudu/127.9.66.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.66.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.66.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.2.keytab.
Entry for principal kudu/127.9.66.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.2.keytab.
I20260925 07:30:05.372186 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790321392278483-9480-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790321392278483-9480-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790321392278483-9480-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790321392278483-9480-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.66.2:0
--local_ip_for_outbound_sockets=127.9.66.2
--webserver_interface=127.9.66.2
--webserver_port=0
--tserver_master_addrs=127.9.66.62:37957
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.2.keytab
--principal=kudu/127.9.66.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:46883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:30:05.450502 12087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:30:05.450695 12087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:30:05.450716 12087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:30:05.451810 12087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:30:05.451850 12087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:30:05.451881 12087 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.66.2
I20260925 07:30:05.453218 12087 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:46883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790321392278483-9480-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790321392278483-9480-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.2.keytab
--principal=kudu/127.9.66.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.66.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790321392278483-9480-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.66.2
--webserver_port=0
--tserver_master_addrs=127.9.66.62:37957
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12087
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.66.2
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790321392278483-9480-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:30:05.453389 12087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:30:05.453572 12087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:30:05.458395 12093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:30:05.458385 12095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:30:05.458504 12092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:30:05.458530 12087 server_base.cc:1034] running on GCE node
I20260925 07:30:05.458858 12087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:30:05.459122 12087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:30:05.460263 12087 hybrid_clock.cc:648] HybridClock initialized: now 1790321405460252 us; error 24 us; skew 500 ppm
Sep 25 07:30:05 dist-test-slave-57fn krb5kdc[11846](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321405, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:30:05.462574 12087 init.cc:377] Logged in from keytab as kudu/127.9.66.2@KRBTEST.COM (short username kudu)
I20260925 07:30:05.463019 12087 webserver.cc:460] Webserver started at http://127.9.66.2:40901/ using document root <none> and password file <none>
I20260925 07:30:05.463248 12087 fs_manager.cc:359] Metadata directory not provided
I20260925 07:30:05.463312 12087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:30:05.463461 12087 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:30:05.464469 12087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790321392278483-9480-0/minicluster-data/ts-1/data/instance:
uuid: "640108fed48e4f1e904773bd55446e91"
format_stamp: "Formatted at 2026-09-25 07:30:05 on dist-test-slave-57fn"
server_key: "e4493231a9a1afe278b120cef2d3941b"
server_key_iv: "f814c1d757a6257b7e863f2a7b92619a"
server_key_version: "encryptionkey@0"
I20260925 07:30:05.464820 12087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790321392278483-9480-0/minicluster-data/ts-1/wal/instance:
uuid: "640108fed48e4f1e904773bd55446e91"
format_stamp: "Formatted at 2026-09-25 07:30:05 on dist-test-slave-57fn"
server_key: "e4493231a9a1afe278b120cef2d3941b"
server_key_iv: "f814c1d757a6257b7e863f2a7b92619a"
server_key_version: "encryptionkey@0"
I20260925 07:30:05.466212 12087 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 07:30:05.466889 12102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:05.467062 12087 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 07:30:05.467155 12087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790321392278483-9480-0/minicluster-data/ts-1/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790321392278483-9480-0/minicluster-data/ts-1/wal
uuid: "640108fed48e4f1e904773bd55446e91"
format_stamp: "Formatted at 2026-09-25 07:30:05 on dist-test-slave-57fn"
server_key: "e4493231a9a1afe278b120cef2d3941b"
server_key_iv: "f814c1d757a6257b7e863f2a7b92619a"
server_key_version: "encryptionkey@0"
I20260925 07:30:05.467231 12087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790321392278483-9480-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790321392278483-9480-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790321392278483-9480-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:30:05.480197 12087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:30:05.482482 12087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:30:05.482640 12087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:30:05.482872 12087 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:30:05.483171 12087 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:30:05.483222 12087 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:05.483261 12087 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:30:05.483289 12087 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:05.489023 12087 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.2:46175
I20260925 07:30:05.489101 12215 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.2:46175 every 8 connection(s)
I20260925 07:30:05.489416 12087 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790321392278483-9480-0/minicluster-data/ts-1/data/info.pb
Sep 25 07:30:05 dist-test-slave-57fn krb5kdc[11846](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321405, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.2@KRBTEST.COM for kudu/127.9.66.62@KRBTEST.COM
I20260925 07:30:05.496625 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 12087
I20260925 07:30:05.496747 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790321392278483-9480-0/minicluster-data/ts-1/wal/instance
I20260925 07:30:05.496968 9480 external_mini_cluster.cc:1428] Setting key ce63181b838b85c8529b0ae4d8f9be31
I20260925 07:30:05.498205 11930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:05.490221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.2:40907 (local address 127.9.66.62:37957)
0925 07:30:05.490290 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:05.490293 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:05.490440 (+ 147us) server_negotiation.cc:407] Connection header received
0925 07:30:05.490872 (+ 432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:05.490873 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:05.490882 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:05.490926 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:30:05.491768 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:05.492249 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:05.493149 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:05.493319 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:05.493367 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:05.495236 (+ 1869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:30:05.495237 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:30:05.495237 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:30:05.495243 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:30:05.496603 (+ 1360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:05.497125 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:05.497127 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:05.497127 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:05.497206 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:05.497470 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:05.497470 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:05.497470 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:05.497631 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:30:05.497708 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:05.498042 (+ 334us) server_negotiation.cc:299] Negotiation successful
0925 07:30:05.498117 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 07:30:05.498397 12218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:05.490180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:37957 (local address 127.9.66.2:40907)
0925 07:30:05.490333 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:05.490366 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 07:30:05.490786 (+ 420us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:30:05.491002 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:30:05.491005 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:30:05.491525 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:30:05.491701 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:05.491705 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:05.492329 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:05.492329 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:05.493083 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:05.493084 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:05.493385 (+ 301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:05.493386 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:05.493481 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:05.493921 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:30:05.493928 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:30:05.495159 (+ 1231us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:30:05.496749 (+ 1590us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:30:05.496756 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:30:05.496758 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:30:05.497046 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:30:05.497262 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:30:05.497263 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:30:05.497263 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:30:05.497408 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:30:05.497699 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:30:05.497705 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:30:05.497846 (+ 141us) client_negotiation.cc:769] Sending connection context
0925 07:30:05.498007 (+ 161us) client_negotiation.cc:240] Negotiation successful
0925 07:30:05.498149 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":84,"threads_started":1}
I20260925 07:30:05.498886 12216 heartbeater.cc:347] Connected to a master server at 127.9.66.62:37957
I20260925 07:30:05.498991 12216 heartbeater.cc:464] Registering TS with master...
I20260925 07:30:05.499153 12216 heartbeater.cc:511] Master 127.9.66.62:37957 requested a full tablet report, sending...
I20260925 07:30:05.499516 11891 ts_manager.cc:194] Registered new tserver with Master: 640108fed48e4f1e904773bd55446e91 (127.9.66.2:46175)
I20260925 07:30:05.499938 11891 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.66.2@KRBTEST.COM'} at 127.9.66.2:40907
WARNING: no policy specified for kudu/127.9.66.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.66.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.66.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.3.keytab.
Entry for principal kudu/127.9.66.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.3.keytab.
I20260925 07:30:05.520632 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790321392278483-9480-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790321392278483-9480-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790321392278483-9480-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790321392278483-9480-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.66.3:0
--local_ip_for_outbound_sockets=127.9.66.3
--webserver_interface=127.9.66.3
--webserver_port=0
--tserver_master_addrs=127.9.66.62:37957
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.3.keytab
--principal=kudu/127.9.66.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:46883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:30:05.599236 12221 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:30:05.599423 12221 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:30:05.599447 12221 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:30:05.600600 12221 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:30:05.600639 12221 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:30:05.600665 12221 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.66.3
I20260925 07:30:05.602135 12221 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:46883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790321392278483-9480-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790321392278483-9480-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.3.keytab
--principal=kudu/127.9.66.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.66.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790321392278483-9480-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.66.3
--webserver_port=0
--tserver_master_addrs=127.9.66.62:37957
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12221
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.66.3
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790321392278483-9480-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:30:05.602370 12221 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:30:05.602579 12221 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:30:05.607308 12226 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:30:05.607302 12229 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:30:05.607357 12227 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:30:05.607710 12221 server_base.cc:1034] running on GCE node
I20260925 07:30:05.607911 12221 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:30:05.608098 12221 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:30:05.609239 12221 hybrid_clock.cc:648] HybridClock initialized: now 1790321405609229 us; error 26 us; skew 500 ppm
Sep 25 07:30:05 dist-test-slave-57fn krb5kdc[11846](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321405, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:30:05.611536 12221 init.cc:377] Logged in from keytab as kudu/127.9.66.3@KRBTEST.COM (short username kudu)
I20260925 07:30:05.611946 12221 webserver.cc:460] Webserver started at http://127.9.66.3:42953/ using document root <none> and password file <none>
I20260925 07:30:05.612176 12221 fs_manager.cc:359] Metadata directory not provided
I20260925 07:30:05.612241 12221 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:30:05.612387 12221 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:30:05.613366 12221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790321392278483-9480-0/minicluster-data/ts-2/data/instance:
uuid: "7ecf9d31e7f5441b9111685a6628ec4a"
format_stamp: "Formatted at 2026-09-25 07:30:05 on dist-test-slave-57fn"
server_key: "9e13281550e34f3d1079ccbc04a6435d"
server_key_iv: "311b4005643d5e97ecc420c3d5ec9bcc"
server_key_version: "encryptionkey@0"
I20260925 07:30:05.613739 12221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790321392278483-9480-0/minicluster-data/ts-2/wal/instance:
uuid: "7ecf9d31e7f5441b9111685a6628ec4a"
format_stamp: "Formatted at 2026-09-25 07:30:05 on dist-test-slave-57fn"
server_key: "9e13281550e34f3d1079ccbc04a6435d"
server_key_iv: "311b4005643d5e97ecc420c3d5ec9bcc"
server_key_version: "encryptionkey@0"
I20260925 07:30:05.615033 12221 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 07:30:05.615764 12236 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:05.615937 12221 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 07:30:05.616035 12221 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790321392278483-9480-0/minicluster-data/ts-2/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790321392278483-9480-0/minicluster-data/ts-2/wal
uuid: "7ecf9d31e7f5441b9111685a6628ec4a"
format_stamp: "Formatted at 2026-09-25 07:30:05 on dist-test-slave-57fn"
server_key: "9e13281550e34f3d1079ccbc04a6435d"
server_key_iv: "311b4005643d5e97ecc420c3d5ec9bcc"
server_key_version: "encryptionkey@0"
I20260925 07:30:05.616111 12221 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790321392278483-9480-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790321392278483-9480-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790321392278483-9480-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:30:05.640782 12221 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:30:05.642627 12221 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:30:05.642786 12221 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:30:05.643015 12221 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:30:05.643326 12221 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:30:05.643380 12221 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:05.643419 12221 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:30:05.643445 12221 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:05.649583 12221 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.3:40711
I20260925 07:30:05.649624 12349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.3:40711 every 8 connection(s)
I20260925 07:30:05.649917 12221 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790321392278483-9480-0/minicluster-data/ts-2/data/info.pb
I20260925 07:30:05.655121 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 12221
I20260925 07:30:05.655220 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790321392278483-9480-0/minicluster-data/ts-2/wal/instance
I20260925 07:30:05.655393 9480 external_mini_cluster.cc:1428] Setting key b439023f7ac965173a53e6962e8c6977
Sep 25 07:30:05 dist-test-slave-57fn krb5kdc[11846](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321405, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.3@KRBTEST.COM for kudu/127.9.66.62@KRBTEST.COM
I20260925 07:30:05.658535 11930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:05.650573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.3:38305 (local address 127.9.66.62:37957)
0925 07:30:05.650610 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:05.650613 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:05.650811 (+ 198us) server_negotiation.cc:407] Connection header received
0925 07:30:05.651368 (+ 557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:05.651370 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:05.651382 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:05.651442 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:30:05.652370 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:05.652875 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:05.653965 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:05.654157 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:05.654227 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:05.656410 (+ 2183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:30:05.656411 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:30:05.656412 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:30:05.656418 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:30:05.657484 (+ 1066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:05.657871 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:05.657872 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:05.657872 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:05.657919 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:05.658106 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:05.658106 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:05.658106 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:05.658191 (+ 85us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:30:05.658235 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:05.658426 (+ 191us) server_negotiation.cc:299] Negotiation successful
0925 07:30:05.658478 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 07:30:05.658684 12352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:05.650564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:37957 (local address 127.9.66.3:38305)
0925 07:30:05.650699 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:05.650732 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 07:30:05.651247 (+ 515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:30:05.651511 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:30:05.651515 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:30:05.652072 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:30:05.652252 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:05.652256 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:05.652968 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:05.652969 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:05.653872 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:05.653875 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:05.654244 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:05.654245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:05.654364 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:05.654745 (+ 381us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:30:05.654752 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:30:05.656335 (+ 1583us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:30:05.657603 (+ 1268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:30:05.657613 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:30:05.657615 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:30:05.657802 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:30:05.657973 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:30:05.657974 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:30:05.657974 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:30:05.658058 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:30:05.658239 (+ 181us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:30:05.658242 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:30:05.658336 (+ 94us) client_negotiation.cc:769] Sending connection context
0925 07:30:05.658415 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 07:30:05.658514 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":66,"threads_started":1}
I20260925 07:30:05.660110 12350 heartbeater.cc:347] Connected to a master server at 127.9.66.62:37957
I20260925 07:30:05.660190 12350 heartbeater.cc:464] Registering TS with master...
I20260925 07:30:05.660305 12350 heartbeater.cc:511] Master 127.9.66.62:37957 requested a full tablet report, sending...
I20260925 07:30:05.660584 11891 ts_manager.cc:194] Registered new tserver with Master: 7ecf9d31e7f5441b9111685a6628ec4a (127.9.66.3:40711)
I20260925 07:30:05.660923 11891 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.66.3@KRBTEST.COM'} at 127.9.66.3:38305
I20260925 07:30:05.666971 9480 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 07:30:05.677348 11930 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:30:05.676526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59072 (local address 127.9.66.62:37957)
0925 07:30:05.676568 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:05.676571 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:05.676688 (+ 117us) server_negotiation.cc:407] Connection header received
0925 07:30:05.676726 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:05.676727 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:05.676735 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:05.676782 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:30:05.677287 (+ 505us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59072: BlockingRecv error: recv got EOF from 127.0.0.1:59072 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 07:30:05.678195 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 11953
I20260925 07:30:05.682808 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 12087
I20260925 07:30:05.686904 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 12221
I20260925 07:30:05.690824 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 11859
2026-09-25T07:30:05Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1815 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790321392278483-9480-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 07:30:05 dist-test-slave-57fn krb5kdc[12362](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 07:30:05 dist-test-slave-57fn krb5kdc[12362](info): set up 2 sockets
Sep 25 07:30:05 dist-test-slave-57fn krb5kdc[12362](info): commencing operation
krb5kdc: starting...
W20260925 07:30:06.713490 9480 mini_kdc.cc:121] Time spent starting KDC: real 1.003s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790321392278483-9480-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790321392278483-9480-0/krb5kdc/test-user.keytab.
Sep 25 07:30:06 dist-test-slave-57fn krb5kdc[12362](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321406, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T07:30:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T07:30:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.66.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.66.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.66.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab.
Entry for principal kudu/127.9.66.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab.
I20260925 07:30:06.826181 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790321392278483-9480-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790321392278483-9480-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790321392278483-9480-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.66.62:33169
--webserver_interface=127.9.66.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab
--principal=kudu/127.9.66.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:40765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.66.62:33169
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:30:06.902050 12375 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:30:06.902312 12375 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:30:06.902350 12375 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:30:06.903451 12375 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:30:06.903522 12375 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:30:06.903545 12375 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:30:06.903571 12375 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:30:06.903601 12375 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:30:06.905015 12375 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:40765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790321392278483-9480-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790321392278483-9480-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.66.62:33169
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab
--principal=kudu/127.9.66.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.66.62:33169
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.66.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12375
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790321392278483-9480-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:30:06.905273 12375 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:30:06.905488 12375 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:30:06.910324 12383 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:30:06.910449 12375 server_base.cc:1034] running on GCE node
W20260925 07:30:06.910334 12380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:30:06.910337 12381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:30:06.910768 12375 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:30:06.911043 12375 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:30:06.912189 12375 hybrid_clock.cc:648] HybridClock initialized: now 1790321406912175 us; error 29 us; skew 500 ppm
Sep 25 07:30:06 dist-test-slave-57fn krb5kdc[12362](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321406, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:30:06.914749 12375 init.cc:377] Logged in from keytab as kudu/127.9.66.62@KRBTEST.COM (short username kudu)
I20260925 07:30:06.915237 12375 webserver.cc:460] Webserver started at http://127.9.66.62:40351/ using document root <none> and password file <none>
I20260925 07:30:06.915462 12375 fs_manager.cc:359] Metadata directory not provided
I20260925 07:30:06.915504 12375 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:30:06.915656 12375 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:30:06.916705 12375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790321392278483-9480-0/minicluster-data/master-0/data/instance:
uuid: "8b0c948ad0334a49af5f52949bfbd258"
format_stamp: "Formatted at 2026-09-25 07:30:06 on dist-test-slave-57fn"
server_key: "632845679670eff39dba05e194ef6b54"
server_key_iv: "40959e129cf5655a0b5e5783ce5c66a5"
server_key_version: "encryptionkey@0"
I20260925 07:30:06.917079 12375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790321392278483-9480-0/minicluster-data/master-0/wal/instance:
uuid: "8b0c948ad0334a49af5f52949bfbd258"
format_stamp: "Formatted at 2026-09-25 07:30:06 on dist-test-slave-57fn"
server_key: "632845679670eff39dba05e194ef6b54"
server_key_iv: "40959e129cf5655a0b5e5783ce5c66a5"
server_key_version: "encryptionkey@0"
I20260925 07:30:06.918431 12375 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 07:30:06.919057 12390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:06.919231 12375 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:06.919299 12375 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790321392278483-9480-0/minicluster-data/master-0/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790321392278483-9480-0/minicluster-data/master-0/wal
uuid: "8b0c948ad0334a49af5f52949bfbd258"
format_stamp: "Formatted at 2026-09-25 07:30:06 on dist-test-slave-57fn"
server_key: "632845679670eff39dba05e194ef6b54"
server_key_iv: "40959e129cf5655a0b5e5783ce5c66a5"
server_key_version: "encryptionkey@0"
I20260925 07:30:06.919384 12375 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790321392278483-9480-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790321392278483-9480-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790321392278483-9480-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:30:06.930711 12375 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:30:06.932885 12375 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:30:06.933043 12375 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:30:06.936668 12375 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.62:33169
I20260925 07:30:06.936694 12442 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.62:33169 every 8 connection(s)
I20260925 07:30:06.937014 12375 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
I20260925 07:30:06.937798 12443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:30:06.939824 12443 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b0c948ad0334a49af5f52949bfbd258: Bootstrap starting.
I20260925 07:30:06.940074 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 12375
I20260925 07:30:06.940171 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790321392278483-9480-0/minicluster-data/master-0/wal/instance
I20260925 07:30:06.940318 9480 external_mini_cluster.cc:1428] Setting key 49026f4dbc5ac5d9b7902fcbbec5417e
I20260925 07:30:06.940424 12443 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8b0c948ad0334a49af5f52949bfbd258: Neither blocks nor log segments found. Creating new log.
I20260925 07:30:06.940755 12443 log.cc:824] T 00000000000000000000000000000000 P 8b0c948ad0334a49af5f52949bfbd258: Log is configured to *not* fsync() on all Append() calls
I20260925 07:30:06.941526 12443 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b0c948ad0334a49af5f52949bfbd258: No bootstrap required, opened a new log
I20260925 07:30:06.943718 12443 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8b0c948ad0334a49af5f52949bfbd258 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b0c948ad0334a49af5f52949bfbd258" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 33169 } }
I20260925 07:30:06.943887 12443 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8b0c948ad0334a49af5f52949bfbd258 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:30:06.943920 12443 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8b0c948ad0334a49af5f52949bfbd258 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b0c948ad0334a49af5f52949bfbd258, State: Initialized, Role: FOLLOWER
I20260925 07:30:06.944037 12443 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b0c948ad0334a49af5f52949bfbd258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b0c948ad0334a49af5f52949bfbd258" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 33169 } }
I20260925 07:30:06.944123 12443 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8b0c948ad0334a49af5f52949bfbd258 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 07:30:06.944154 12443 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8b0c948ad0334a49af5f52949bfbd258 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 07:30:06.944199 12443 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b0c948ad0334a49af5f52949bfbd258 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:30:06.944875 12443 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8b0c948ad0334a49af5f52949bfbd258 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b0c948ad0334a49af5f52949bfbd258" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 33169 } }
I20260925 07:30:06.944977 12443 leader_election.cc:302] T 00000000000000000000000000000000 P 8b0c948ad0334a49af5f52949bfbd258 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8b0c948ad0334a49af5f52949bfbd258; no voters:
I20260925 07:30:06.945158 12443 leader_election.cc:288] T 00000000000000000000000000000000 P 8b0c948ad0334a49af5f52949bfbd258 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 07:30:06.945179 12448 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8b0c948ad0334a49af5f52949bfbd258 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:30:06.945294 12448 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8b0c948ad0334a49af5f52949bfbd258 [term 1 LEADER]: Becoming Leader. State: Replica: 8b0c948ad0334a49af5f52949bfbd258, State: Running, Role: LEADER
I20260925 07:30:06.945370 12448 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b0c948ad0334a49af5f52949bfbd258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b0c948ad0334a49af5f52949bfbd258" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 33169 } }
I20260925 07:30:06.945400 12443 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8b0c948ad0334a49af5f52949bfbd258 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:30:06.945737 12449 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b0c948ad0334a49af5f52949bfbd258 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8b0c948ad0334a49af5f52949bfbd258" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b0c948ad0334a49af5f52949bfbd258" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 33169 } } }
I20260925 07:30:06.945736 12450 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b0c948ad0334a49af5f52949bfbd258 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b0c948ad0334a49af5f52949bfbd258. Latest consensus state: current_term: 1 leader_uuid: "8b0c948ad0334a49af5f52949bfbd258" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b0c948ad0334a49af5f52949bfbd258" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 33169 } } }
I20260925 07:30:06.945866 12449 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b0c948ad0334a49af5f52949bfbd258 [sys.catalog]: This master's current role is: LEADER
I20260925 07:30:06.945874 12450 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b0c948ad0334a49af5f52949bfbd258 [sys.catalog]: This master's current role is: LEADER
I20260925 07:30:06.946728 12455 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 07:30:06.947345 12455 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 25 07:30:06 dist-test-slave-57fn krb5kdc[12362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321406, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.66.62@KRBTEST.COM
I20260925 07:30:06.948712 12455 catalog_manager.cc:1347] Generated new cluster ID: a36964fa8f51479e913d6a79202a9abe
I20260925 07:30:06.948760 12455 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 07:30:06.952342 12447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:06.940875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32836 (local address 127.9.66.62:33169)
0925 07:30:06.941048 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:06.941057 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:06.941095 (+ 38us) server_negotiation.cc:407] Connection header received
0925 07:30:06.941703 (+ 608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:06.941710 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:06.941766 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:06.941902 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:30:06.942520 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:06.943943 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:06.945066 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:06.945713 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:06.946270 (+ 557us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:06.948646 (+ 2376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:30:06.948649 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:30:06.948650 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:30:06.948659 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:30:06.950592 (+ 1933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:06.950976 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:06.950979 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:06.950979 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:06.951079 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:06.951316 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:06.951317 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:06.951317 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:06.951622 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:30:06.951766 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:06.951885 (+ 119us) server_negotiation.cc:299] Negotiation successful
0925 07:30:06.952120 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":94,"threads_started":1}
I20260925 07:30:06.953101 9480 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 07:30:06.970322 12447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:06.966340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32838 (local address 127.9.66.62:33169)
0925 07:30:06.966395 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:06.966397 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:06.966509 (+ 112us) server_negotiation.cc:407] Connection header received
0925 07:30:06.966569 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:06.966569 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:06.966579 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:06.966636 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:30:06.967005 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:06.967741 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:06.968770 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:06.969071 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:06.969126 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:06.969884 (+ 758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:30:06.969886 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:30:06.969886 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 07:30:06.970072 (+ 186us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:30:06.970087 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 07:30:06.970092 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:30:06.970145 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:06.970225 (+ 80us) server_negotiation.cc:299] Negotiation successful
0925 07:30:06.970247 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 07:30:06.979475 12455 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 07:30:06.979954 12455 catalog_manager.cc:1504] Loading token signing keys...
I20260925 07:30:06.989887 12455 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8b0c948ad0334a49af5f52949bfbd258: Generated new TSK 0
I20260925 07:30:06.990067 12455 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 07:30:06.996973 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 12375
2026-09-25T07:30:06Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1304 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790321392278483-9480-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 07:30:07 dist-test-slave-57fn krb5kdc[12476](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 07:30:07 dist-test-slave-57fn krb5kdc[12476](info): set up 2 sockets
Sep 25 07:30:07 dist-test-slave-57fn krb5kdc[12476](info): commencing operation
krb5kdc: starting...
W20260925 07:30:08.013278 9480 mini_kdc.cc:121] Time spent starting KDC: real 0.998s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790321392278483-9480-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790321392278483-9480-0/krb5kdc/test-user.keytab.
Sep 25 07:30:08 dist-test-slave-57fn krb5kdc[12476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321408, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T07:30:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T07:30:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.66.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.66.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.66.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab.
Entry for principal kudu/127.9.66.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab.
I20260925 07:30:08.136739 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790321392278483-9480-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790321392278483-9480-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790321392278483-9480-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.66.62:41149
--webserver_interface=127.9.66.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab
--principal=kudu/127.9.66.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:37935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.66.62:41149
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:30:08.217892 12489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:30:08.218081 12489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:30:08.218103 12489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:30:08.219267 12489 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:30:08.219318 12489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:30:08.219333 12489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:30:08.219347 12489 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:30:08.219359 12489 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:30:08.220824 12489 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:37935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790321392278483-9480-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790321392278483-9480-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.66.62:41149
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab
--principal=kudu/127.9.66.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.66.62:41149
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.66.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12489
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790321392278483-9480-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:30:08.221068 12489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:30:08.221292 12489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:30:08.226402 12495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:30:08.226287 12497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:30:08.226289 12494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:30:08.226511 12489 server_base.cc:1034] running on GCE node
I20260925 07:30:08.226815 12489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:30:08.227054 12489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:30:08.228197 12489 hybrid_clock.cc:648] HybridClock initialized: now 1790321408228169 us; error 44 us; skew 500 ppm
Sep 25 07:30:08 dist-test-slave-57fn krb5kdc[12476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321408, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:30:08.230670 12489 init.cc:377] Logged in from keytab as kudu/127.9.66.62@KRBTEST.COM (short username kudu)
I20260925 07:30:08.231119 12489 webserver.cc:460] Webserver started at http://127.9.66.62:43943/ using document root <none> and password file <none>
I20260925 07:30:08.231370 12489 fs_manager.cc:359] Metadata directory not provided
I20260925 07:30:08.231416 12489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:30:08.231578 12489 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:30:08.232648 12489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790321392278483-9480-0/minicluster-data/master-0/data/instance:
uuid: "9ef79a242aed44a3ab2d51c96467048c"
format_stamp: "Formatted at 2026-09-25 07:30:08 on dist-test-slave-57fn"
server_key: "7f7e4e4d96175cbc7ddd10727d2615d6"
server_key_iv: "bc8a514ed199a939e6c9523c8cdd0ded"
server_key_version: "encryptionkey@0"
I20260925 07:30:08.233009 12489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790321392278483-9480-0/minicluster-data/master-0/wal/instance:
uuid: "9ef79a242aed44a3ab2d51c96467048c"
format_stamp: "Formatted at 2026-09-25 07:30:08 on dist-test-slave-57fn"
server_key: "7f7e4e4d96175cbc7ddd10727d2615d6"
server_key_iv: "bc8a514ed199a939e6c9523c8cdd0ded"
server_key_version: "encryptionkey@0"
I20260925 07:30:08.234357 12489 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 07:30:08.235073 12504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:08.235252 12489 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 07:30:08.235337 12489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790321392278483-9480-0/minicluster-data/master-0/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790321392278483-9480-0/minicluster-data/master-0/wal
uuid: "9ef79a242aed44a3ab2d51c96467048c"
format_stamp: "Formatted at 2026-09-25 07:30:08 on dist-test-slave-57fn"
server_key: "7f7e4e4d96175cbc7ddd10727d2615d6"
server_key_iv: "bc8a514ed199a939e6c9523c8cdd0ded"
server_key_version: "encryptionkey@0"
I20260925 07:30:08.235419 12489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790321392278483-9480-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790321392278483-9480-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790321392278483-9480-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:30:08.248143 12489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:30:08.250489 12489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:30:08.250653 12489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:30:08.254468 12489 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.62:41149
I20260925 07:30:08.254529 12556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.62:41149 every 8 connection(s)
I20260925 07:30:08.254828 12489 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
I20260925 07:30:08.255841 12557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:30:08.257972 12557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ef79a242aed44a3ab2d51c96467048c: Bootstrap starting.
I20260925 07:30:08.258486 12557 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9ef79a242aed44a3ab2d51c96467048c: Neither blocks nor log segments found. Creating new log.
I20260925 07:30:08.258706 12557 log.cc:824] T 00000000000000000000000000000000 P 9ef79a242aed44a3ab2d51c96467048c: Log is configured to *not* fsync() on all Append() calls
I20260925 07:30:08.259325 12557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ef79a242aed44a3ab2d51c96467048c: No bootstrap required, opened a new log
I20260925 07:30:08.260771 12557 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9ef79a242aed44a3ab2d51c96467048c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ef79a242aed44a3ab2d51c96467048c" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 41149 } }
I20260925 07:30:08.260900 12557 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9ef79a242aed44a3ab2d51c96467048c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:30:08.260921 12557 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9ef79a242aed44a3ab2d51c96467048c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ef79a242aed44a3ab2d51c96467048c, State: Initialized, Role: FOLLOWER
I20260925 07:30:08.261056 12557 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ef79a242aed44a3ab2d51c96467048c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ef79a242aed44a3ab2d51c96467048c" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 41149 } }
I20260925 07:30:08.261133 12557 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9ef79a242aed44a3ab2d51c96467048c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 07:30:08.261155 12557 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9ef79a242aed44a3ab2d51c96467048c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 07:30:08.261183 12557 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9ef79a242aed44a3ab2d51c96467048c [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:30:08.261396 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 12489
I20260925 07:30:08.261500 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790321392278483-9480-0/minicluster-data/master-0/wal/instance
I20260925 07:30:08.261708 9480 external_mini_cluster.cc:1428] Setting key 55546467bc3d769657f73a58570c3ffc
I20260925 07:30:08.262323 12557 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9ef79a242aed44a3ab2d51c96467048c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ef79a242aed44a3ab2d51c96467048c" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 41149 } }
I20260925 07:30:08.262454 12557 leader_election.cc:302] T 00000000000000000000000000000000 P 9ef79a242aed44a3ab2d51c96467048c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9ef79a242aed44a3ab2d51c96467048c; no voters:
I20260925 07:30:08.262601 12557 leader_election.cc:288] T 00000000000000000000000000000000 P 9ef79a242aed44a3ab2d51c96467048c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 07:30:08.262657 12562 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9ef79a242aed44a3ab2d51c96467048c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:30:08.262856 12562 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9ef79a242aed44a3ab2d51c96467048c [term 1 LEADER]: Becoming Leader. State: Replica: 9ef79a242aed44a3ab2d51c96467048c, State: Running, Role: LEADER
I20260925 07:30:08.262893 12557 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9ef79a242aed44a3ab2d51c96467048c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:30:08.262943 12562 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9ef79a242aed44a3ab2d51c96467048c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ef79a242aed44a3ab2d51c96467048c" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 41149 } }
I20260925 07:30:08.263238 12562 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ef79a242aed44a3ab2d51c96467048c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9ef79a242aed44a3ab2d51c96467048c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ef79a242aed44a3ab2d51c96467048c" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 41149 } } }
I20260925 07:30:08.263379 12562 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ef79a242aed44a3ab2d51c96467048c [sys.catalog]: This master's current role is: LEADER
I20260925 07:30:08.263554 12562 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ef79a242aed44a3ab2d51c96467048c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ef79a242aed44a3ab2d51c96467048c. Latest consensus state: current_term: 1 leader_uuid: "9ef79a242aed44a3ab2d51c96467048c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ef79a242aed44a3ab2d51c96467048c" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 41149 } } }
I20260925 07:30:08.263625 12562 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ef79a242aed44a3ab2d51c96467048c [sys.catalog]: This master's current role is: LEADER
I20260925 07:30:08.263729 12571 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 07:30:08.264238 12571 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 07:30:08.266076 12571 catalog_manager.cc:1347] Generated new cluster ID: 95eb3260a8df4a01ac4606b926081f10
I20260925 07:30:08.266127 12571 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 07:30:08 dist-test-slave-57fn krb5kdc[12476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321408, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.66.62@KRBTEST.COM
I20260925 07:30:08.272178 12561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:08.262203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45126 (local address 127.9.66.62:41149)
0925 07:30:08.262376 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:08.262386 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:08.262457 (+ 71us) server_negotiation.cc:407] Connection header received
0925 07:30:08.262566 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:08.262572 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:08.262625 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:08.262795 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:30:08.263533 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:08.265006 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:08.266016 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:08.266460 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:08.266587 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:08.268844 (+ 2257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:30:08.268847 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:30:08.268848 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:30:08.268858 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:30:08.270619 (+ 1761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:08.271036 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:08.271039 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:08.271039 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:08.271148 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:08.271367 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:08.271367 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:08.271368 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:08.271595 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:30:08.271781 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:08.271848 (+ 67us) server_negotiation.cc:299] Negotiation successful
0925 07:30:08.271985 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":129,"thread_start_us":83,"threads_started":1}
I20260925 07:30:08.289425 12571 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 07:30:08.290045 12571 catalog_manager.cc:1504] Loading token signing keys...
I20260925 07:30:08.293440 12571 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9ef79a242aed44a3ab2d51c96467048c: Generated new TSK 0
I20260925 07:30:08.293644 12571 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.66.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.66.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.66.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.1.keytab.
Entry for principal kudu/127.9.66.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.1.keytab.
I20260925 07:30:08.326617 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790321392278483-9480-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790321392278483-9480-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790321392278483-9480-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790321392278483-9480-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.66.1:0
--local_ip_for_outbound_sockets=127.9.66.1
--webserver_interface=127.9.66.1
--webserver_port=0
--tserver_master_addrs=127.9.66.62:41149
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.1.keytab
--principal=kudu/127.9.66.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:37935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:30:08.407261 12583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:30:08.407485 12583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:30:08.407508 12583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:30:08.408682 12583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:30:08.408728 12583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:30:08.408754 12583 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.66.1
I20260925 07:30:08.410267 12583 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:37935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790321392278483-9480-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790321392278483-9480-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.1.keytab
--principal=kudu/127.9.66.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.66.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790321392278483-9480-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.66.1
--webserver_port=0
--tserver_master_addrs=127.9.66.62:41149
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12583
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.66.1
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790321392278483-9480-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:30:08.410506 12583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:30:08.410722 12583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:30:08.415520 12589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:30:08.415527 12588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:30:08.415618 12591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:30:08.415709 12583 server_base.cc:1034] running on GCE node
I20260925 07:30:08.415984 12583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:30:08.416232 12583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:30:08.417407 12583 hybrid_clock.cc:648] HybridClock initialized: now 1790321408417403 us; error 49 us; skew 500 ppm
Sep 25 07:30:08 dist-test-slave-57fn krb5kdc[12476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321408, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:30:08.419839 12583 init.cc:377] Logged in from keytab as kudu/127.9.66.1@KRBTEST.COM (short username kudu)
I20260925 07:30:08.420334 12583 webserver.cc:460] Webserver started at http://127.9.66.1:34739/ using document root <none> and password file <none>
I20260925 07:30:08.420549 12583 fs_manager.cc:359] Metadata directory not provided
I20260925 07:30:08.420614 12583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:30:08.420760 12583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:30:08.421833 12583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790321392278483-9480-0/minicluster-data/ts-0/data/instance:
uuid: "a10006633a9b459998f06a10de3d6450"
format_stamp: "Formatted at 2026-09-25 07:30:08 on dist-test-slave-57fn"
server_key: "dd2ef175c9640771882a58e9553e9f07"
server_key_iv: "681b1720d415aad12c823eb98fecb333"
server_key_version: "encryptionkey@0"
I20260925 07:30:08.422173 12583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790321392278483-9480-0/minicluster-data/ts-0/wal/instance:
uuid: "a10006633a9b459998f06a10de3d6450"
format_stamp: "Formatted at 2026-09-25 07:30:08 on dist-test-slave-57fn"
server_key: "dd2ef175c9640771882a58e9553e9f07"
server_key_iv: "681b1720d415aad12c823eb98fecb333"
server_key_version: "encryptionkey@0"
I20260925 07:30:08.423668 12583 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 07:30:08.424402 12598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:08.424578 12583 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 07:30:08.424688 12583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790321392278483-9480-0/minicluster-data/ts-0/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790321392278483-9480-0/minicluster-data/ts-0/wal
uuid: "a10006633a9b459998f06a10de3d6450"
format_stamp: "Formatted at 2026-09-25 07:30:08 on dist-test-slave-57fn"
server_key: "dd2ef175c9640771882a58e9553e9f07"
server_key_iv: "681b1720d415aad12c823eb98fecb333"
server_key_version: "encryptionkey@0"
I20260925 07:30:08.424808 12583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790321392278483-9480-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790321392278483-9480-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790321392278483-9480-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:30:08.438910 12583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:30:08.441262 12583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:30:08.441425 12583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:30:08.441715 12583 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:30:08.442055 12583 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:30:08.442112 12583 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:08.442154 12583 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:30:08.442183 12583 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:08.448235 12583 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.1:38301
I20260925 07:30:08.448278 12711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.1:38301 every 8 connection(s)
I20260925 07:30:08.448633 12583 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790321392278483-9480-0/minicluster-data/ts-0/data/info.pb
I20260925 07:30:08.451071 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 12583
I20260925 07:30:08.451195 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790321392278483-9480-0/minicluster-data/ts-0/wal/instance
I20260925 07:30:08.451391 9480 external_mini_cluster.cc:1428] Setting key f704db5fe34e2d5ba20072c37f14b52d
Sep 25 07:30:08 dist-test-slave-57fn krb5kdc[12476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321408, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.1@KRBTEST.COM for kudu/127.9.66.62@KRBTEST.COM
I20260925 07:30:08.458122 12561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:08.449401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.1:40557 (local address 127.9.66.62:41149)
0925 07:30:08.449448 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:08.449450 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:08.449682 (+ 232us) server_negotiation.cc:407] Connection header received
0925 07:30:08.450148 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:08.450149 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:08.450161 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:08.450221 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:30:08.451195 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:08.451858 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:08.452862 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:08.453081 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:08.453169 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:08.455284 (+ 2115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:30:08.455285 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:30:08.455286 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:30:08.455301 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:30:08.456719 (+ 1418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:08.457114 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:08.457115 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:08.457115 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:08.457171 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:08.457442 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:08.457443 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:08.457443 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:08.457602 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:30:08.457677 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:08.457950 (+ 273us) server_negotiation.cc:299] Negotiation successful
0925 07:30:08.458034 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 07:30:08.458320 12714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:08.449358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:41149 (local address 127.9.66.1:40557)
0925 07:30:08.449542 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:08.449584 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 07:30:08.450065 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:30:08.450293 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:30:08.450298 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:30:08.450878 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:30:08.451086 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:08.451091 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:08.451953 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:08.451954 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:08.452745 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:08.452747 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:08.453179 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:08.453179 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:08.453299 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:08.453884 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:30:08.453891 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:30:08.455179 (+ 1288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:30:08.456812 (+ 1633us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:30:08.456818 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:30:08.456820 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:30:08.457035 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:30:08.457245 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:30:08.457245 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:30:08.457246 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:30:08.457356 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:30:08.457682 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:30:08.457687 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:30:08.457824 (+ 137us) client_negotiation.cc:769] Sending connection context
0925 07:30:08.457929 (+ 105us) client_negotiation.cc:240] Negotiation successful
0925 07:30:08.458078 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":87,"threads_started":1}
I20260925 07:30:08.458806 12712 heartbeater.cc:347] Connected to a master server at 127.9.66.62:41149
I20260925 07:30:08.458904 12712 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.9.66.2@KRBTEST.COM; defaulting to no policy
I20260925 07:30:08.459084 12712 heartbeater.cc:511] Master 127.9.66.62:41149 requested a full tablet report, sending...
I20260925 07:30:08.459597 12521 ts_manager.cc:194] Registered new tserver with Master: a10006633a9b459998f06a10de3d6450 (127.9.66.1:38301)
I20260925 07:30:08.460187 12521 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.66.1@KRBTEST.COM'} at 127.9.66.1:40557
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.66.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.66.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.2.keytab.
Entry for principal kudu/127.9.66.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.2.keytab.
I20260925 07:30:08.476109 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790321392278483-9480-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790321392278483-9480-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790321392278483-9480-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790321392278483-9480-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.66.2:0
--local_ip_for_outbound_sockets=127.9.66.2
--webserver_interface=127.9.66.2
--webserver_port=0
--tserver_master_addrs=127.9.66.62:41149
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.2.keytab
--principal=kudu/127.9.66.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:37935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:30:08.561947 12717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:30:08.562219 12717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:30:08.562261 12717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:30:08.563777 12717 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:30:08.563818 12717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:30:08.563846 12717 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.66.2
I20260925 07:30:08.566097 12717 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:37935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790321392278483-9480-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790321392278483-9480-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.2.keytab
--principal=kudu/127.9.66.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.66.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790321392278483-9480-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.66.2
--webserver_port=0
--tserver_master_addrs=127.9.66.62:41149
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12717
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.66.2
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790321392278483-9480-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:30:08.566378 12717 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:30:08.566632 12717 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:30:08.571983 12725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:30:08.572021 12722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:30:08.571997 12723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:30:08.572273 12717 server_base.cc:1034] running on GCE node
I20260925 07:30:08.572485 12717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:30:08.572775 12717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:30:08.573977 12717 hybrid_clock.cc:648] HybridClock initialized: now 1790321408573956 us; error 35 us; skew 500 ppm
Sep 25 07:30:08 dist-test-slave-57fn krb5kdc[12476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321408, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:30:08.577039 12717 init.cc:377] Logged in from keytab as kudu/127.9.66.2@KRBTEST.COM (short username kudu)
I20260925 07:30:08.577466 12717 webserver.cc:460] Webserver started at http://127.9.66.2:45499/ using document root <none> and password file <none>
I20260925 07:30:08.577733 12717 fs_manager.cc:359] Metadata directory not provided
I20260925 07:30:08.577805 12717 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:30:08.577945 12717 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:30:08.579443 12717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790321392278483-9480-0/minicluster-data/ts-1/data/instance:
uuid: "ece1b0d1b8d14f65a25910b259c78b15"
format_stamp: "Formatted at 2026-09-25 07:30:08 on dist-test-slave-57fn"
server_key: "47a7195c61403fc0b010580e93165bac"
server_key_iv: "f02d264761c04e1ea512378ec3b708e7"
server_key_version: "encryptionkey@0"
I20260925 07:30:08.579867 12717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790321392278483-9480-0/minicluster-data/ts-1/wal/instance:
uuid: "ece1b0d1b8d14f65a25910b259c78b15"
format_stamp: "Formatted at 2026-09-25 07:30:08 on dist-test-slave-57fn"
server_key: "47a7195c61403fc0b010580e93165bac"
server_key_iv: "f02d264761c04e1ea512378ec3b708e7"
server_key_version: "encryptionkey@0"
I20260925 07:30:08.581540 12717 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260925 07:30:08.582540 12732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:08.582705 12717 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:30:08.582811 12717 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790321392278483-9480-0/minicluster-data/ts-1/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790321392278483-9480-0/minicluster-data/ts-1/wal
uuid: "ece1b0d1b8d14f65a25910b259c78b15"
format_stamp: "Formatted at 2026-09-25 07:30:08 on dist-test-slave-57fn"
server_key: "47a7195c61403fc0b010580e93165bac"
server_key_iv: "f02d264761c04e1ea512378ec3b708e7"
server_key_version: "encryptionkey@0"
I20260925 07:30:08.582890 12717 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790321392278483-9480-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790321392278483-9480-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790321392278483-9480-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:30:08.617764 12717 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:30:08.620090 12717 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:30:08.620196 12717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:30:08.620383 12717 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:30:08.620688 12717 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:30:08.620724 12717 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:08.620747 12717 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:30:08.620798 12717 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:08.626626 12717 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.2:36903
I20260925 07:30:08.626664 12845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.2:36903 every 8 connection(s)
I20260925 07:30:08.626962 12717 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790321392278483-9480-0/minicluster-data/ts-1/data/info.pb
I20260925 07:30:08.631224 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 12717
I20260925 07:30:08.631337 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790321392278483-9480-0/minicluster-data/ts-1/wal/instance
I20260925 07:30:08.631531 9480 external_mini_cluster.cc:1428] Setting key 6d8d33764b6a15ea9a3a7224b93c7186
Sep 25 07:30:08 dist-test-slave-57fn krb5kdc[12476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321408, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.2@KRBTEST.COM for kudu/127.9.66.62@KRBTEST.COM
I20260925 07:30:08.636701 12561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:08.627716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.2:33013 (local address 127.9.66.62:41149)
0925 07:30:08.627777 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:08.627780 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:08.628000 (+ 220us) server_negotiation.cc:407] Connection header received
0925 07:30:08.628517 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:08.628518 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:08.628528 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:08.628583 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:30:08.629441 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:08.630004 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:08.631054 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:08.631376 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:08.631459 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:08.633902 (+ 2443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:30:08.633903 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:30:08.633904 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:30:08.633911 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:30:08.635235 (+ 1324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:08.635725 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:08.635726 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:08.635726 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:08.635786 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:08.636065 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:08.636066 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:08.636066 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:08.636185 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:30:08.636235 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:08.636579 (+ 344us) server_negotiation.cc:299] Negotiation successful
0925 07:30:08.636639 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 07:30:08.637032 12848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:08.627710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:41149 (local address 127.9.66.2:33013)
0925 07:30:08.627865 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:08.627906 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 07:30:08.628420 (+ 514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:30:08.628660 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:30:08.628663 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:30:08.629180 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:30:08.629367 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:08.629371 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:08.630082 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:08.630082 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:08.630963 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:08.630965 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:08.631480 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:08.631481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:08.631621 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:08.632150 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:30:08.632159 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:30:08.633805 (+ 1646us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:30:08.635329 (+ 1524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:30:08.635335 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:30:08.635337 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:30:08.635644 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:30:08.635859 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:30:08.635861 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:30:08.635861 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:30:08.635999 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:30:08.636267 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:30:08.636274 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:30:08.636442 (+ 168us) client_negotiation.cc:769] Sending connection context
0925 07:30:08.636564 (+ 122us) client_negotiation.cc:240] Negotiation successful
0925 07:30:08.636741 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":89,"threads_started":1}
I20260925 07:30:08.637631 12846 heartbeater.cc:347] Connected to a master server at 127.9.66.62:41149
I20260925 07:30:08.637718 12846 heartbeater.cc:464] Registering TS with master...
I20260925 07:30:08.637905 12846 heartbeater.cc:511] Master 127.9.66.62:41149 requested a full tablet report, sending...
I20260925 07:30:08.638329 12521 ts_manager.cc:194] Registered new tserver with Master: ece1b0d1b8d14f65a25910b259c78b15 (127.9.66.2:36903)
I20260925 07:30:08.638720 12521 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.66.2@KRBTEST.COM'} at 127.9.66.2:33013
WARNING: no policy specified for kudu/127.9.66.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.66.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.66.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.3.keytab.
Entry for principal kudu/127.9.66.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.3.keytab.
I20260925 07:30:08.657104 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790321392278483-9480-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790321392278483-9480-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790321392278483-9480-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790321392278483-9480-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.66.3:0
--local_ip_for_outbound_sockets=127.9.66.3
--webserver_interface=127.9.66.3
--webserver_port=0
--tserver_master_addrs=127.9.66.62:41149
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.3.keytab
--principal=kudu/127.9.66.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:37935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:30:08.735880 12851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:30:08.736105 12851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:30:08.736142 12851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:30:08.737310 12851 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:30:08.737377 12851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:30:08.737422 12851 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.66.3
I20260925 07:30:08.738857 12851 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:37935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790321392278483-9480-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790321392278483-9480-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.3.keytab
--principal=kudu/127.9.66.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.66.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790321392278483-9480-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.66.3
--webserver_port=0
--tserver_master_addrs=127.9.66.62:41149
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12851
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.66.3
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790321392278483-9480-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:30:08.739100 12851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:30:08.739333 12851 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:30:08.744262 12857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:30:08.744234 12859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:30:08.744263 12856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:30:08.744383 12851 server_base.cc:1034] running on GCE node
I20260925 07:30:08.744653 12851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:30:08.744896 12851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:30:08.746068 12851 hybrid_clock.cc:648] HybridClock initialized: now 1790321408746041 us; error 37 us; skew 500 ppm
Sep 25 07:30:08 dist-test-slave-57fn krb5kdc[12476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321408, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:30:08.748505 12851 init.cc:377] Logged in from keytab as kudu/127.9.66.3@KRBTEST.COM (short username kudu)
I20260925 07:30:08.748977 12851 webserver.cc:460] Webserver started at http://127.9.66.3:45759/ using document root <none> and password file <none>
I20260925 07:30:08.749210 12851 fs_manager.cc:359] Metadata directory not provided
I20260925 07:30:08.749279 12851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:30:08.749428 12851 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:30:08.750499 12851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790321392278483-9480-0/minicluster-data/ts-2/data/instance:
uuid: "30e1323a669d4f84a388d05532fc652f"
format_stamp: "Formatted at 2026-09-25 07:30:08 on dist-test-slave-57fn"
server_key: "c86ad5f223dbde0c78eca4b27d0e47ff"
server_key_iv: "0410faca412683474bd35893086a432c"
server_key_version: "encryptionkey@0"
I20260925 07:30:08.750865 12851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790321392278483-9480-0/minicluster-data/ts-2/wal/instance:
uuid: "30e1323a669d4f84a388d05532fc652f"
format_stamp: "Formatted at 2026-09-25 07:30:08 on dist-test-slave-57fn"
server_key: "c86ad5f223dbde0c78eca4b27d0e47ff"
server_key_iv: "0410faca412683474bd35893086a432c"
server_key_version: "encryptionkey@0"
I20260925 07:30:08.752141 12851 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 07:30:08.752856 12866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:08.753022 12851 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 07:30:08.753122 12851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790321392278483-9480-0/minicluster-data/ts-2/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790321392278483-9480-0/minicluster-data/ts-2/wal
uuid: "30e1323a669d4f84a388d05532fc652f"
format_stamp: "Formatted at 2026-09-25 07:30:08 on dist-test-slave-57fn"
server_key: "c86ad5f223dbde0c78eca4b27d0e47ff"
server_key_iv: "0410faca412683474bd35893086a432c"
server_key_version: "encryptionkey@0"
I20260925 07:30:08.753202 12851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790321392278483-9480-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790321392278483-9480-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790321392278483-9480-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:30:08.761814 12851 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:30:08.763811 12851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:30:08.763957 12851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:30:08.764246 12851 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:30:08.764605 12851 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:30:08.764654 12851 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:08.764680 12851 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:30:08.764700 12851 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:08.770568 12851 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.3:42149
I20260925 07:30:08.770612 12979 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.3:42149 every 8 connection(s)
I20260925 07:30:08.770902 12851 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790321392278483-9480-0/minicluster-data/ts-2/data/info.pb
I20260925 07:30:08.771219 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 12851
I20260925 07:30:08.771337 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790321392278483-9480-0/minicluster-data/ts-2/wal/instance
I20260925 07:30:08.771523 9480 external_mini_cluster.cc:1428] Setting key e240ffd809f1f42652c68e9857246dd5
Sep 25 07:30:08 dist-test-slave-57fn krb5kdc[12476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321408, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.3@KRBTEST.COM for kudu/127.9.66.62@KRBTEST.COM
I20260925 07:30:08.779430 12561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:08.771597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.3:35309 (local address 127.9.66.62:41149)
0925 07:30:08.771650 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:08.771652 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:08.771900 (+ 248us) server_negotiation.cc:407] Connection header received
0925 07:30:08.772531 (+ 631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:08.772532 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:08.772542 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:08.772585 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:30:08.773444 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:08.773981 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:08.774882 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:08.775048 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:08.775096 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:08.777131 (+ 2035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:30:08.777132 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:30:08.777132 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:30:08.777137 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:30:08.778366 (+ 1229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:08.778720 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:08.778721 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:08.778721 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:08.778764 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:08.778958 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:08.778959 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:08.778959 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:08.779056 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:30:08.779103 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:08.779318 (+ 215us) server_negotiation.cc:299] Negotiation successful
0925 07:30:08.779370 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 07:30:08.779589 12982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:08.771583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:41149 (local address 127.9.66.3:35309)
0925 07:30:08.771728 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:08.771800 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0925 07:30:08.772433 (+ 633us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:30:08.772643 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:30:08.772646 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:30:08.773190 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:30:08.773372 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:08.773376 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:08.774053 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:08.774053 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:08.774812 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:08.774814 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:08.775116 (+ 302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:08.775117 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:08.775207 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:08.775738 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:30:08.775745 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:30:08.777051 (+ 1306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:30:08.778433 (+ 1382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:30:08.778438 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:30:08.778440 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:30:08.778658 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:30:08.778820 (+ 162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:30:08.778820 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:30:08.778821 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:30:08.778908 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:30:08.779106 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:30:08.779110 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:30:08.779209 (+ 99us) client_negotiation.cc:769] Sending connection context
0925 07:30:08.779283 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 07:30:08.779399 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":77,"threads_started":1}
I20260925 07:30:08.780033 12980 heartbeater.cc:347] Connected to a master server at 127.9.66.62:41149
I20260925 07:30:08.780108 12980 heartbeater.cc:464] Registering TS with master...
I20260925 07:30:08.780239 12980 heartbeater.cc:511] Master 127.9.66.62:41149 requested a full tablet report, sending...
I20260925 07:30:08.780555 12521 ts_manager.cc:194] Registered new tserver with Master: 30e1323a669d4f84a388d05532fc652f (127.9.66.3:42149)
I20260925 07:30:08.780889 12521 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.66.3@KRBTEST.COM'} at 127.9.66.3:35309
I20260925 07:30:08.783030 9480 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 07:30:08 dist-test-slave-57fn krb5kdc[12476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321408, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 25 07:30:08 dist-test-slave-57fn krb5kdc[12476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321408, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.9.66.62@KRBTEST.COM
I20260925 07:30:08.804764 12561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:08.798007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45138 (local address 127.9.66.62:41149)
0925 07:30:08.798094 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:08.798096 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:08.798184 (+ 88us) server_negotiation.cc:407] Connection header received
0925 07:30:08.798269 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:08.798270 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:08.798279 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:08.798336 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:30:08.798926 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:08.799401 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:08.800148 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:08.800355 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:08.800434 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:08.802553 (+ 2119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:30:08.802554 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:30:08.802554 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:30:08.802561 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:30:08.803630 (+ 1069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:08.803978 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:08.803979 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:08.803980 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:08.804033 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:08.804226 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:08.804227 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:08.804228 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:08.804385 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:30:08.804485 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:08.804587 (+ 102us) server_negotiation.cc:299] Negotiation successful
0925 07:30:08.804662 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260925 07:30:08.804983 12521 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:45138
I20260925 07:30:08.805352 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 12583
I20260925 07:30:08.809912 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 12717
I20260925 07:30:08.814409 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 12851
I20260925 07:30:08.818349 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 12489
2026-09-25T07:30:08Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1822 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790321392278483-9480-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 07:30:08 dist-test-slave-57fn krb5kdc[12992](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 07:30:08 dist-test-slave-57fn krb5kdc[12992](info): set up 2 sockets
Sep 25 07:30:08 dist-test-slave-57fn krb5kdc[12992](info): commencing operation
krb5kdc: starting...
W20260925 07:30:09.845333 9480 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790321392278483-9480-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790321392278483-9480-0/krb5kdc/test-user.keytab.
Sep 25 07:30:09 dist-test-slave-57fn krb5kdc[12992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321409, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T07:30:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T07:30:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.66.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.66.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.66.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab.
Entry for principal kudu/127.9.66.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab.
I20260925 07:30:09.962338 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790321392278483-9480-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790321392278483-9480-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790321392278483-9480-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.66.62:44597
--webserver_interface=127.9.66.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab
--principal=kudu/127.9.66.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:39973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.66.62:44597
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:30:10.043365 13005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:30:10.043581 13005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:30:10.043605 13005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:30:10.044762 13005 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:30:10.044795 13005 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:30:10.044809 13005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:30:10.044821 13005 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:30:10.044833 13005 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:30:10.046340 13005 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:39973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790321392278483-9480-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790321392278483-9480-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.66.62:44597
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab
--principal=kudu/127.9.66.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.66.62:44597
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.66.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13005
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790321392278483-9480-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:30:10.046527 13005 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:30:10.046692 13005 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:30:10.051594 13010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:30:10.051620 13011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:30:10.051597 13013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:30:10.051644 13005 server_base.cc:1034] running on GCE node
I20260925 07:30:10.052067 13005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:30:10.052388 13005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:30:10.053574 13005 hybrid_clock.cc:648] HybridClock initialized: now 1790321410053569 us; error 38 us; skew 500 ppm
Sep 25 07:30:10 dist-test-slave-57fn krb5kdc[12992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321410, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:30:10.056046 13005 init.cc:377] Logged in from keytab as kudu/127.9.66.62@KRBTEST.COM (short username kudu)
I20260925 07:30:10.056546 13005 webserver.cc:460] Webserver started at http://127.9.66.62:44529/ using document root <none> and password file <none>
I20260925 07:30:10.056788 13005 fs_manager.cc:359] Metadata directory not provided
I20260925 07:30:10.056856 13005 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:30:10.056985 13005 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:30:10.058092 13005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790321392278483-9480-0/minicluster-data/master-0/data/instance:
uuid: "079d16d738774133aef0b0ff9dc9a607"
format_stamp: "Formatted at 2026-09-25 07:30:10 on dist-test-slave-57fn"
server_key: "a85ab484c7f1dddc3984b4febac70514"
server_key_iv: "0ed9666e27358a77f202bc8d793d44e3"
server_key_version: "encryptionkey@0"
I20260925 07:30:10.058472 13005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790321392278483-9480-0/minicluster-data/master-0/wal/instance:
uuid: "079d16d738774133aef0b0ff9dc9a607"
format_stamp: "Formatted at 2026-09-25 07:30:10 on dist-test-slave-57fn"
server_key: "a85ab484c7f1dddc3984b4febac70514"
server_key_iv: "0ed9666e27358a77f202bc8d793d44e3"
server_key_version: "encryptionkey@0"
I20260925 07:30:10.059795 13005 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 07:30:10.060556 13020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:10.060724 13005 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 07:30:10.060823 13005 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790321392278483-9480-0/minicluster-data/master-0/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790321392278483-9480-0/minicluster-data/master-0/wal
uuid: "079d16d738774133aef0b0ff9dc9a607"
format_stamp: "Formatted at 2026-09-25 07:30:10 on dist-test-slave-57fn"
server_key: "a85ab484c7f1dddc3984b4febac70514"
server_key_iv: "0ed9666e27358a77f202bc8d793d44e3"
server_key_version: "encryptionkey@0"
I20260925 07:30:10.060906 13005 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790321392278483-9480-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790321392278483-9480-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790321392278483-9480-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:30:10.101976 13005 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:30:10.104408 13005 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:30:10.104583 13005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:30:10.108732 13005 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.62:44597
I20260925 07:30:10.108784 13072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.62:44597 every 8 connection(s)
I20260925 07:30:10.109326 13005 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
I20260925 07:30:10.110062 13073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:30:10.112298 13073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 079d16d738774133aef0b0ff9dc9a607: Bootstrap starting.
I20260925 07:30:10.112896 13073 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 079d16d738774133aef0b0ff9dc9a607: Neither blocks nor log segments found. Creating new log.
I20260925 07:30:10.113205 13073 log.cc:824] T 00000000000000000000000000000000 P 079d16d738774133aef0b0ff9dc9a607: Log is configured to *not* fsync() on all Append() calls
I20260925 07:30:10.113905 13073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 079d16d738774133aef0b0ff9dc9a607: No bootstrap required, opened a new log
I20260925 07:30:10.115474 13073 raft_consensus.cc:348] T 00000000000000000000000000000000 P 079d16d738774133aef0b0ff9dc9a607 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "079d16d738774133aef0b0ff9dc9a607" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 44597 } }
I20260925 07:30:10.115612 13073 raft_consensus.cc:374] T 00000000000000000000000000000000 P 079d16d738774133aef0b0ff9dc9a607 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:30:10.115634 13073 raft_consensus.cc:729] T 00000000000000000000000000000000 P 079d16d738774133aef0b0ff9dc9a607 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 079d16d738774133aef0b0ff9dc9a607, State: Initialized, Role: FOLLOWER
I20260925 07:30:10.115803 13073 consensus_queue.cc:260] T 00000000000000000000000000000000 P 079d16d738774133aef0b0ff9dc9a607 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "079d16d738774133aef0b0ff9dc9a607" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 44597 } }
I20260925 07:30:10.115875 13073 raft_consensus.cc:388] T 00000000000000000000000000000000 P 079d16d738774133aef0b0ff9dc9a607 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 07:30:10.115898 13073 raft_consensus.cc:482] T 00000000000000000000000000000000 P 079d16d738774133aef0b0ff9dc9a607 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 07:30:10.115952 13073 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 079d16d738774133aef0b0ff9dc9a607 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:30:10.116849 13073 raft_consensus.cc:504] T 00000000000000000000000000000000 P 079d16d738774133aef0b0ff9dc9a607 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "079d16d738774133aef0b0ff9dc9a607" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 44597 } }
I20260925 07:30:10.116962 13073 leader_election.cc:302] T 00000000000000000000000000000000 P 079d16d738774133aef0b0ff9dc9a607 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 079d16d738774133aef0b0ff9dc9a607; no voters:
I20260925 07:30:10.117163 13073 leader_election.cc:288] T 00000000000000000000000000000000 P 079d16d738774133aef0b0ff9dc9a607 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 07:30:10.117184 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 13005
I20260925 07:30:10.117213 13076 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 079d16d738774133aef0b0ff9dc9a607 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:30:10.117302 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790321392278483-9480-0/minicluster-data/master-0/wal/instance
I20260925 07:30:10.117342 13076 raft_consensus.cc:686] T 00000000000000000000000000000000 P 079d16d738774133aef0b0ff9dc9a607 [term 1 LEADER]: Becoming Leader. State: Replica: 079d16d738774133aef0b0ff9dc9a607, State: Running, Role: LEADER
I20260925 07:30:10.117424 13076 consensus_queue.cc:237] T 00000000000000000000000000000000 P 079d16d738774133aef0b0ff9dc9a607 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "079d16d738774133aef0b0ff9dc9a607" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 44597 } }
I20260925 07:30:10.117468 13073 sys_catalog.cc:564] T 00000000000000000000000000000000 P 079d16d738774133aef0b0ff9dc9a607 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:30:10.117496 9480 external_mini_cluster.cc:1428] Setting key 82709eaeeddbf7f613ae9ed490ed2f3e
I20260925 07:30:10.117748 13076 sys_catalog.cc:455] T 00000000000000000000000000000000 P 079d16d738774133aef0b0ff9dc9a607 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 079d16d738774133aef0b0ff9dc9a607. Latest consensus state: current_term: 1 leader_uuid: "079d16d738774133aef0b0ff9dc9a607" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "079d16d738774133aef0b0ff9dc9a607" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 44597 } } }
I20260925 07:30:10.117825 13076 sys_catalog.cc:458] T 00000000000000000000000000000000 P 079d16d738774133aef0b0ff9dc9a607 [sys.catalog]: This master's current role is: LEADER
I20260925 07:30:10.118065 13080 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 07:30:10.118599 13080 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 07:30:10.118610 13078 sys_catalog.cc:455] T 00000000000000000000000000000000 P 079d16d738774133aef0b0ff9dc9a607 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "079d16d738774133aef0b0ff9dc9a607" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "079d16d738774133aef0b0ff9dc9a607" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 44597 } } }
I20260925 07:30:10.118896 13078 sys_catalog.cc:458] T 00000000000000000000000000000000 P 079d16d738774133aef0b0ff9dc9a607 [sys.catalog]: This master's current role is: LEADER
I20260925 07:30:10.120348 13080 catalog_manager.cc:1347] Generated new cluster ID: dc29d9ad1e5442fab63df3fbf517e832
I20260925 07:30:10.120401 13080 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 07:30:10.125993 13080 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 07:30:10.126462 13080 catalog_manager.cc:1504] Loading token signing keys...
Sep 25 07:30:10 dist-test-slave-57fn krb5kdc[12992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321409, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.66.62@KRBTEST.COM
I20260925 07:30:10.132591 13080 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 079d16d738774133aef0b0ff9dc9a607: Generated new TSK 0
I20260925 07:30:10.132755 13080 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 07:30:10.134125 13088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:10.118268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56020 (local address 127.9.66.62:44597)
0925 07:30:10.118521 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:10.118532 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:10.118803 (+ 271us) server_negotiation.cc:407] Connection header received
0925 07:30:10.118929 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:10.118937 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:10.119014 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:10.119140 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:30:10.120156 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:10.121177 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:10.122076 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:10.122513 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:10.122630 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:10.130375 (+ 7745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:30:10.130379 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:30:10.130380 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:30:10.130393 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:30:10.132390 (+ 1997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:10.132863 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:10.132866 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:10.132866 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:10.132972 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:10.133173 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:10.133174 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:10.133174 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:10.133461 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:30:10.133627 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:10.133732 (+ 105us) server_negotiation.cc:299] Negotiation successful
0925 07:30:10.133893 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":63,"threads_started":1}
WARNING: no policy specified for kudu/127.9.66.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.66.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.66.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.1.keytab.
Entry for principal kudu/127.9.66.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.1.keytab.
I20260925 07:30:10.161228 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790321392278483-9480-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790321392278483-9480-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790321392278483-9480-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790321392278483-9480-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.66.1:0
--local_ip_for_outbound_sockets=127.9.66.1
--webserver_interface=127.9.66.1
--webserver_port=0
--tserver_master_addrs=127.9.66.62:44597
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.1.keytab
--principal=kudu/127.9.66.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:39973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:30:10.242625 13099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:30:10.242835 13099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:30:10.242858 13099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:30:10.244002 13099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:30:10.244041 13099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:30:10.244067 13099 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.66.1
I20260925 07:30:10.245608 13099 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:39973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790321392278483-9480-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790321392278483-9480-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.1.keytab
--principal=kudu/127.9.66.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.66.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790321392278483-9480-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.66.1
--webserver_port=0
--tserver_master_addrs=127.9.66.62:44597
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13099
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.66.1
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790321392278483-9480-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:30:10.245843 13099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:30:10.246088 13099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:30:10.251124 13107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:30:10.251087 13105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:30:10.251127 13104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:30:10.251374 13099 server_base.cc:1034] running on GCE node
I20260925 07:30:10.251605 13099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:30:10.251900 13099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:30:10.253064 13099 hybrid_clock.cc:648] HybridClock initialized: now 1790321410253053 us; error 30 us; skew 500 ppm
Sep 25 07:30:10 dist-test-slave-57fn krb5kdc[12992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321410, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:30:10.255740 13099 init.cc:377] Logged in from keytab as kudu/127.9.66.1@KRBTEST.COM (short username kudu)
I20260925 07:30:10.256238 13099 webserver.cc:460] Webserver started at http://127.9.66.1:37083/ using document root <none> and password file <none>
I20260925 07:30:10.256469 13099 fs_manager.cc:359] Metadata directory not provided
I20260925 07:30:10.256536 13099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:30:10.256682 13099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:30:10.257900 13099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790321392278483-9480-0/minicluster-data/ts-0/data/instance:
uuid: "781a76b584244799b003dc5cc39b357e"
format_stamp: "Formatted at 2026-09-25 07:30:10 on dist-test-slave-57fn"
server_key: "7fe210c3af8741fc7ad45195c6ffcc9a"
server_key_iv: "e754b72c5966bd54b80026b9204e4a21"
server_key_version: "encryptionkey@0"
I20260925 07:30:10.258265 13099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790321392278483-9480-0/minicluster-data/ts-0/wal/instance:
uuid: "781a76b584244799b003dc5cc39b357e"
format_stamp: "Formatted at 2026-09-25 07:30:10 on dist-test-slave-57fn"
server_key: "7fe210c3af8741fc7ad45195c6ffcc9a"
server_key_iv: "e754b72c5966bd54b80026b9204e4a21"
server_key_version: "encryptionkey@0"
I20260925 07:30:10.259573 13099 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 07:30:10.260313 13114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:10.260499 13099 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 07:30:10.260596 13099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790321392278483-9480-0/minicluster-data/ts-0/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790321392278483-9480-0/minicluster-data/ts-0/wal
uuid: "781a76b584244799b003dc5cc39b357e"
format_stamp: "Formatted at 2026-09-25 07:30:10 on dist-test-slave-57fn"
server_key: "7fe210c3af8741fc7ad45195c6ffcc9a"
server_key_iv: "e754b72c5966bd54b80026b9204e4a21"
server_key_version: "encryptionkey@0"
I20260925 07:30:10.260679 13099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790321392278483-9480-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790321392278483-9480-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790321392278483-9480-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:30:10.271438 13099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:30:10.273663 13099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:30:10.273828 13099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:30:10.274067 13099 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:30:10.274365 13099 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:30:10.274420 13099 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:10.274459 13099 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:30:10.274485 13099 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:10.280411 13099 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.1:45745
I20260925 07:30:10.280462 13227 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.1:45745 every 8 connection(s)
I20260925 07:30:10.280752 13099 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790321392278483-9480-0/minicluster-data/ts-0/data/info.pb
I20260925 07:30:10.286298 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 13099
I20260925 07:30:10.286412 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790321392278483-9480-0/minicluster-data/ts-0/wal/instance
I20260925 07:30:10.286618 9480 external_mini_cluster.cc:1428] Setting key 55c83ae985ad6bd650fe7bbfecd5e6b0
Sep 25 07:30:10 dist-test-slave-57fn krb5kdc[12992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321410, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.1@KRBTEST.COM for kudu/127.9.66.62@KRBTEST.COM
I20260925 07:30:10.291550 13088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:10.281500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.1:41413 (local address 127.9.66.62:44597)
0925 07:30:10.281597 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:10.281599 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:10.281769 (+ 170us) server_negotiation.cc:407] Connection header received
0925 07:30:10.282729 (+ 960us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:10.282730 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:10.282746 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:10.282813 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:30:10.283529 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:10.284253 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:10.285891 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:10.286234 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:10.286314 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:10.288489 (+ 2175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:30:10.288491 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:30:10.288491 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:30:10.288499 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:30:10.290157 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:10.290584 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:10.290586 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:10.290586 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:10.290644 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:10.290912 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:10.290913 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:10.290913 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:10.291027 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:30:10.291098 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:10.291377 (+ 279us) server_negotiation.cc:299] Negotiation successful
0925 07:30:10.291464 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 07:30:10.291738 13230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:10.281458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:44597 (local address 127.9.66.1:41413)
0925 07:30:10.281644 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:10.281690 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 07:30:10.282609 (+ 919us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:30:10.282883 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:30:10.282887 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:30:10.283136 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:30:10.283423 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:10.283429 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:10.284364 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:10.284365 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:10.285784 (+ 1419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:10.285787 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:10.286388 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:10.286390 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:10.286540 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:10.287056 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:30:10.287065 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:30:10.288391 (+ 1326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:30:10.290265 (+ 1874us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:30:10.290272 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:30:10.290274 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:30:10.290503 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:30:10.290719 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:30:10.290720 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:30:10.290720 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:30:10.290836 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:30:10.291104 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:30:10.291111 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:30:10.291243 (+ 132us) client_negotiation.cc:769] Sending connection context
0925 07:30:10.291351 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 07:30:10.291502 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":114,"threads_started":1}
I20260925 07:30:10.292176 13228 heartbeater.cc:347] Connected to a master server at 127.9.66.62:44597
I20260925 07:30:10.292266 13228 heartbeater.cc:464] Registering TS with master...
I20260925 07:30:10.292454 13228 heartbeater.cc:511] Master 127.9.66.62:44597 requested a full tablet report, sending...
I20260925 07:30:10.292955 13037 ts_manager.cc:194] Registered new tserver with Master: 781a76b584244799b003dc5cc39b357e (127.9.66.1:45745)
I20260925 07:30:10.293663 13037 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.66.1@KRBTEST.COM'} at 127.9.66.1:41413
WARNING: no policy specified for kudu/127.9.66.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.66.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.66.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.2.keytab.
Entry for principal kudu/127.9.66.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.2.keytab.
I20260925 07:30:10.312470 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790321392278483-9480-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790321392278483-9480-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790321392278483-9480-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790321392278483-9480-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.66.2:0
--local_ip_for_outbound_sockets=127.9.66.2
--webserver_interface=127.9.66.2
--webserver_port=0
--tserver_master_addrs=127.9.66.62:44597
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.2.keytab
--principal=kudu/127.9.66.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:39973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:30:10.392755 13233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:30:10.392987 13233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:30:10.393010 13233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:30:10.394179 13233 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:30:10.394220 13233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:30:10.394246 13233 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.66.2
I20260925 07:30:10.395678 13233 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:39973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790321392278483-9480-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790321392278483-9480-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.2.keytab
--principal=kudu/127.9.66.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.66.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790321392278483-9480-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.66.2
--webserver_port=0
--tserver_master_addrs=127.9.66.62:44597
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13233
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.66.2
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790321392278483-9480-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:30:10.395958 13233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:30:10.396189 13233 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:30:10.401319 13239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:30:10.401284 13238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:30:10.401286 13233 server_base.cc:1034] running on GCE node
W20260925 07:30:10.401436 13241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:30:10.401684 13233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:30:10.401957 13233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:30:10.403126 13233 hybrid_clock.cc:648] HybridClock initialized: now 1790321410403113 us; error 31 us; skew 500 ppm
Sep 25 07:30:10 dist-test-slave-57fn krb5kdc[12992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321410, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:30:10.405498 13233 init.cc:377] Logged in from keytab as kudu/127.9.66.2@KRBTEST.COM (short username kudu)
I20260925 07:30:10.406015 13233 webserver.cc:460] Webserver started at http://127.9.66.2:38627/ using document root <none> and password file <none>
I20260925 07:30:10.406250 13233 fs_manager.cc:359] Metadata directory not provided
I20260925 07:30:10.406329 13233 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:30:10.406479 13233 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:30:10.407511 13233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790321392278483-9480-0/minicluster-data/ts-1/data/instance:
uuid: "316a41b23e5d422fb3503538d2bc3682"
format_stamp: "Formatted at 2026-09-25 07:30:10 on dist-test-slave-57fn"
server_key: "c7e2045bcdcd4171306f4d8c587f97de"
server_key_iv: "cd68fbf79f3a993bd9233f55c059dc20"
server_key_version: "encryptionkey@0"
I20260925 07:30:10.407879 13233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790321392278483-9480-0/minicluster-data/ts-1/wal/instance:
uuid: "316a41b23e5d422fb3503538d2bc3682"
format_stamp: "Formatted at 2026-09-25 07:30:10 on dist-test-slave-57fn"
server_key: "c7e2045bcdcd4171306f4d8c587f97de"
server_key_iv: "cd68fbf79f3a993bd9233f55c059dc20"
server_key_version: "encryptionkey@0"
I20260925 07:30:10.409165 13233 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 07:30:10.409948 13248 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:10.410131 13233 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:10.410234 13233 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790321392278483-9480-0/minicluster-data/ts-1/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790321392278483-9480-0/minicluster-data/ts-1/wal
uuid: "316a41b23e5d422fb3503538d2bc3682"
format_stamp: "Formatted at 2026-09-25 07:30:10 on dist-test-slave-57fn"
server_key: "c7e2045bcdcd4171306f4d8c587f97de"
server_key_iv: "cd68fbf79f3a993bd9233f55c059dc20"
server_key_version: "encryptionkey@0"
I20260925 07:30:10.410326 13233 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790321392278483-9480-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790321392278483-9480-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790321392278483-9480-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:30:10.427553 13233 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:30:10.429371 13233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:30:10.429587 13233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:30:10.429813 13233 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:30:10.430127 13233 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:30:10.430184 13233 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:10.430226 13233 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:30:10.430254 13233 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:10.436048 13233 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.2:46731
I20260925 07:30:10.436088 13361 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.2:46731 every 8 connection(s)
I20260925 07:30:10.436407 13233 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790321392278483-9480-0/minicluster-data/ts-1/data/info.pb
I20260925 07:30:10.436950 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 13233
I20260925 07:30:10.437042 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790321392278483-9480-0/minicluster-data/ts-1/wal/instance
I20260925 07:30:10.437292 9480 external_mini_cluster.cc:1428] Setting key edc82e71e7e76b5b1a4567a67255bdf4
Sep 25 07:30:10 dist-test-slave-57fn krb5kdc[12992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321410, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.2@KRBTEST.COM for kudu/127.9.66.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.66.3@KRBTEST.COM; defaulting to no policy
I20260925 07:30:10.446378 13088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:10.437200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.2:46661 (local address 127.9.66.62:44597)
0925 07:30:10.437248 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:10.437251 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:10.437425 (+ 174us) server_negotiation.cc:407] Connection header received
0925 07:30:10.438540 (+ 1115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:10.438541 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:10.438557 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:10.438620 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:30:10.439291 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:10.439790 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:10.440781 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:10.440995 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:10.441085 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:10.443449 (+ 2364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:30:10.443451 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:30:10.443451 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:30:10.443459 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:30:10.444863 (+ 1404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:10.445355 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:10.445357 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:10.445357 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:10.445420 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:10.445709 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:10.445710 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:10.445710 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:10.445833 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:30:10.445914 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:10.446187 (+ 273us) server_negotiation.cc:299] Negotiation successful
0925 07:30:10.446272 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 07:30:10.446552 13364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:10.437101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:44597 (local address 127.9.66.2:46661)
0925 07:30:10.437293 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:10.437333 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 07:30:10.438412 (+ 1079us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:30:10.438682 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:30:10.438685 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:30:10.438948 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:30:10.439191 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:10.439195 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:10.439897 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:10.439898 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:10.440684 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:10.440686 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:10.441095 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:10.441096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:10.441232 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:10.441769 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:30:10.441791 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:30:10.443347 (+ 1556us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:30:10.444980 (+ 1633us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:30:10.444987 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:30:10.444989 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:30:10.445278 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:30:10.445492 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:30:10.445492 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:30:10.445493 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:30:10.445636 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:30:10.445905 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:30:10.445911 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:30:10.446051 (+ 140us) client_negotiation.cc:769] Sending connection context
0925 07:30:10.446155 (+ 104us) client_negotiation.cc:240] Negotiation successful
0925 07:30:10.446317 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":118,"threads_started":1}
I20260925 07:30:10.446976 13362 heartbeater.cc:347] Connected to a master server at 127.9.66.62:44597
I20260925 07:30:10.447095 13362 heartbeater.cc:464] Registering TS with master...
I20260925 07:30:10.447270 13362 heartbeater.cc:511] Master 127.9.66.62:44597 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.66.3@KRBTEST.COM" created.
I20260925 07:30:10.447716 13037 ts_manager.cc:194] Registered new tserver with Master: 316a41b23e5d422fb3503538d2bc3682 (127.9.66.2:46731)
I20260925 07:30:10.448196 13037 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.66.2@KRBTEST.COM'} at 127.9.66.2:46661
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.66.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.3.keytab.
Entry for principal kudu/127.9.66.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.3.keytab.
I20260925 07:30:10.464846 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790321392278483-9480-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790321392278483-9480-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790321392278483-9480-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790321392278483-9480-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.66.3:0
--local_ip_for_outbound_sockets=127.9.66.3
--webserver_interface=127.9.66.3
--webserver_port=0
--tserver_master_addrs=127.9.66.62:44597
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.3.keytab
--principal=kudu/127.9.66.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:39973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:30:10.547580 13367 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:30:10.547833 13367 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:30:10.547871 13367 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:30:10.549022 13367 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:30:10.549096 13367 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:30:10.549141 13367 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.66.3
I20260925 07:30:10.550675 13367 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:39973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790321392278483-9480-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790321392278483-9480-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.3.keytab
--principal=kudu/127.9.66.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.66.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790321392278483-9480-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.66.3
--webserver_port=0
--tserver_master_addrs=127.9.66.62:44597
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13367
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.66.3
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790321392278483-9480-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:30:10.550956 13367 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:30:10.551188 13367 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:30:10.556214 13372 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:30:10.556236 13373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:30:10.556291 13367 server_base.cc:1034] running on GCE node
W20260925 07:30:10.556236 13375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:30:10.556663 13367 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:30:10.556887 13367 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:30:10.558029 13367 hybrid_clock.cc:648] HybridClock initialized: now 1790321410558008 us; error 37 us; skew 500 ppm
Sep 25 07:30:10 dist-test-slave-57fn krb5kdc[12992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321410, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:30:10.560539 13367 init.cc:377] Logged in from keytab as kudu/127.9.66.3@KRBTEST.COM (short username kudu)
I20260925 07:30:10.560978 13367 webserver.cc:460] Webserver started at http://127.9.66.3:37353/ using document root <none> and password file <none>
I20260925 07:30:10.561180 13367 fs_manager.cc:359] Metadata directory not provided
I20260925 07:30:10.561226 13367 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:30:10.561314 13367 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:30:10.562362 13367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790321392278483-9480-0/minicluster-data/ts-2/data/instance:
uuid: "1db0ccfe30db4a60affe1a5b4a25f81b"
format_stamp: "Formatted at 2026-09-25 07:30:10 on dist-test-slave-57fn"
server_key: "ce5a8a4e72d61efc7aebeeb7817eeeb9"
server_key_iv: "35853936706679867eaa28d7d600d93c"
server_key_version: "encryptionkey@0"
I20260925 07:30:10.562654 13367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790321392278483-9480-0/minicluster-data/ts-2/wal/instance:
uuid: "1db0ccfe30db4a60affe1a5b4a25f81b"
format_stamp: "Formatted at 2026-09-25 07:30:10 on dist-test-slave-57fn"
server_key: "ce5a8a4e72d61efc7aebeeb7817eeeb9"
server_key_iv: "35853936706679867eaa28d7d600d93c"
server_key_version: "encryptionkey@0"
I20260925 07:30:10.564164 13367 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 07:30:10.565008 13382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:10.565176 13367 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 07:30:10.565272 13367 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790321392278483-9480-0/minicluster-data/ts-2/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790321392278483-9480-0/minicluster-data/ts-2/wal
uuid: "1db0ccfe30db4a60affe1a5b4a25f81b"
format_stamp: "Formatted at 2026-09-25 07:30:10 on dist-test-slave-57fn"
server_key: "ce5a8a4e72d61efc7aebeeb7817eeeb9"
server_key_iv: "35853936706679867eaa28d7d600d93c"
server_key_version: "encryptionkey@0"
I20260925 07:30:10.565356 13367 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790321392278483-9480-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790321392278483-9480-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790321392278483-9480-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:30:10.602465 13367 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:30:10.604878 13367 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:30:10.605075 13367 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:30:10.605326 13367 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:30:10.605751 13367 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:30:10.605805 13367 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:10.605847 13367 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:30:10.605873 13367 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:10.611830 13367 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.3:42653
I20260925 07:30:10.611873 13495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.3:42653 every 8 connection(s)
I20260925 07:30:10.612186 13367 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790321392278483-9480-0/minicluster-data/ts-2/data/info.pb
Sep 25 07:30:10 dist-test-slave-57fn krb5kdc[12992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321410, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.3@KRBTEST.COM for kudu/127.9.66.62@KRBTEST.COM
I20260925 07:30:10.619849 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 13367
I20260925 07:30:10.619966 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790321392278483-9480-0/minicluster-data/ts-2/wal/instance
I20260925 07:30:10.620182 9480 external_mini_cluster.cc:1428] Setting key e470a06458fc34d650c1c49dab54c493
I20260925 07:30:10.621665 13088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:10.612963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.3:60267 (local address 127.9.66.62:44597)
0925 07:30:10.613039 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:10.613041 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:10.613285 (+ 244us) server_negotiation.cc:407] Connection header received
0925 07:30:10.614283 (+ 998us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:10.614284 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:10.614294 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:10.614348 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:30:10.614903 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:10.615385 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:10.616339 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:10.616512 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:10.616559 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:10.618814 (+ 2255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:30:10.618815 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:30:10.618816 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:30:10.618822 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:30:10.620238 (+ 1416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:10.620706 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:10.620707 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:10.620708 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:10.620772 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:10.621059 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:10.621060 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:10.621060 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:10.621195 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:30:10.621264 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:10.621480 (+ 216us) server_negotiation.cc:299] Negotiation successful
0925 07:30:10.621597 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 07:30:10.621907 13498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:10.612964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:44597 (local address 127.9.66.3:60267)
0925 07:30:10.613155 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:10.613190 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 07:30:10.614191 (+ 1001us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:30:10.614399 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:30:10.614402 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:30:10.614644 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:30:10.614836 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:10.614839 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:10.615465 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:10.615465 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:10.616268 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:10.616270 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:10.616580 (+ 310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:10.616581 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:10.616673 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:10.617220 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:30:10.617228 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:30:10.618731 (+ 1503us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:30:10.620322 (+ 1591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:30:10.620328 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:30:10.620330 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:30:10.620598 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:30:10.620858 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:30:10.620859 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:30:10.620859 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:30:10.620991 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:30:10.621262 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:30:10.621266 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:30:10.621376 (+ 110us) client_negotiation.cc:769] Sending connection context
0925 07:30:10.621459 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 07:30:10.621639 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":98,"threads_started":1}
I20260925 07:30:10.622393 13496 heartbeater.cc:347] Connected to a master server at 127.9.66.62:44597
I20260925 07:30:10.622509 13496 heartbeater.cc:464] Registering TS with master...
I20260925 07:30:10.622680 13496 heartbeater.cc:511] Master 127.9.66.62:44597 requested a full tablet report, sending...
I20260925 07:30:10.623046 13037 ts_manager.cc:194] Registered new tserver with Master: 1db0ccfe30db4a60affe1a5b4a25f81b (127.9.66.3:42653)
I20260925 07:30:10.623376 13037 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.66.3@KRBTEST.COM'} at 127.9.66.3:60267
I20260925 07:30:10.631861 9480 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 07:30:10 dist-test-slave-57fn krb5kdc[12992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321410, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 07:30:10 dist-test-slave-57fn krb5kdc[12992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321410, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.66.1@KRBTEST.COM
I20260925 07:30:10.654366 13503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:10.646781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43958 (local address 127.9.66.1:45745)
0925 07:30:10.646933 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:10.646939 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:10.646956 (+ 17us) server_negotiation.cc:407] Connection header received
0925 07:30:10.646980 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:10.646982 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:10.646995 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:10.647061 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:30:10.647666 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:10.648401 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:10.649120 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:10.649413 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:10.649467 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:10.651695 (+ 2228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:30:10.651703 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:30:10.651704 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:30:10.651710 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:30:10.653302 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:10.653715 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:10.653717 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:10.653717 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:10.653770 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:10.653944 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:10.653944 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:10.653944 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:10.654051 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:30:10.654141 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:10.654230 (+ 89us) server_negotiation.cc:299] Negotiation successful
0925 07:30:10.654290 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":89,"threads_started":1}
Sep 25 07:30:10 dist-test-slave-57fn krb5kdc[12992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321410, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.66.62@KRBTEST.COM
I20260925 07:30:10.662205 13088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:10.655613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56026 (local address 127.9.66.62:44597)
0925 07:30:10.655688 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:10.655689 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:10.655713 (+ 24us) server_negotiation.cc:407] Connection header received
0925 07:30:10.655767 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:10.655767 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:10.655776 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:10.655831 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:30:10.656233 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:10.656709 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:10.657397 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:10.657697 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:10.657756 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:10.659811 (+ 2055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:30:10.659812 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:30:10.659813 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:30:10.659819 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:30:10.661160 (+ 1341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:10.661481 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:10.661482 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:10.661482 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:10.661589 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:10.661776 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:10.661777 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:10.661777 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:10.661907 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:30:10.661979 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:10.662064 (+ 85us) server_negotiation.cc:299] Negotiation successful
0925 07:30:10.662116 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
W20260925 07:30:10.662456 13037 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56026
I20260925 07:30:10.662830 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 13099
I20260925 07:30:10.667811 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 13233
I20260925 07:30:10.672185 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 13367
I20260925 07:30:10.676352 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 13005
2026-09-25T07:30:10Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1858 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790321392278483-9480-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 07:30:10 dist-test-slave-57fn krb5kdc[13508](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 07:30:10 dist-test-slave-57fn krb5kdc[13508](info): set up 2 sockets
Sep 25 07:30:10 dist-test-slave-57fn krb5kdc[13508](info): commencing operation
krb5kdc: starting...
W20260925 07:30:11.713476 9480 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790321392278483-9480-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790321392278483-9480-0/krb5kdc/test-user.keytab.
Sep 25 07:30:11 dist-test-slave-57fn krb5kdc[13508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321411, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T07:30:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T07:30:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.66.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.66.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.66.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab.
Entry for principal kudu/127.9.66.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab.
I20260925 07:30:11.833843 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790321392278483-9480-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790321392278483-9480-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790321392278483-9480-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.66.62:44971
--webserver_interface=127.9.66.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab
--principal=kudu/127.9.66.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:40037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.66.62:44971
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:30:11.911695 13521 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:30:11.911893 13521 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:30:11.911916 13521 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:30:11.912971 13521 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:30:11.913003 13521 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:30:11.913017 13521 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:30:11.913029 13521 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:30:11.913040 13521 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:30:11.914446 13521 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:40037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790321392278483-9480-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790321392278483-9480-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.66.62:44971
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab
--principal=kudu/127.9.66.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.66.62:44971
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.66.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13521
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790321392278483-9480-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:30:11.914619 13521 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:30:11.914783 13521 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:30:11.919365 13527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:30:11.919394 13529 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:30:11.919449 13521 server_base.cc:1034] running on GCE node
W20260925 07:30:11.919386 13526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:30:11.919766 13521 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:30:11.920039 13521 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:30:11.921200 13521 hybrid_clock.cc:648] HybridClock initialized: now 1790321411921188 us; error 28 us; skew 500 ppm
Sep 25 07:30:11 dist-test-slave-57fn krb5kdc[13508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321411, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:30:11.923738 13521 init.cc:377] Logged in from keytab as kudu/127.9.66.62@KRBTEST.COM (short username kudu)
I20260925 07:30:11.924134 13521 fs_manager.cc:359] Metadata directory not provided
I20260925 07:30:11.924206 13521 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:30:11.924342 13521 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:30:11.925379 13521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790321392278483-9480-0/minicluster-data/master-0/data/instance:
uuid: "ef9eed146bc14a37a5428c146316db1d"
format_stamp: "Formatted at 2026-09-25 07:30:11 on dist-test-slave-57fn"
server_key: "a0fbe17807098d9f6c52c9e623c1f2fe"
server_key_iv: "df88895d74cbb567e3527d20a739562a"
server_key_version: "encryptionkey@0"
I20260925 07:30:11.925766 13521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790321392278483-9480-0/minicluster-data/master-0/wal/instance:
uuid: "ef9eed146bc14a37a5428c146316db1d"
format_stamp: "Formatted at 2026-09-25 07:30:11 on dist-test-slave-57fn"
server_key: "a0fbe17807098d9f6c52c9e623c1f2fe"
server_key_iv: "df88895d74cbb567e3527d20a739562a"
server_key_version: "encryptionkey@0"
I20260925 07:30:11.927011 13521 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:30:11.927753 13535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:11.928014 13521 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:30:11.928112 13521 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790321392278483-9480-0/minicluster-data/master-0/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790321392278483-9480-0/minicluster-data/master-0/wal
uuid: "ef9eed146bc14a37a5428c146316db1d"
format_stamp: "Formatted at 2026-09-25 07:30:11 on dist-test-slave-57fn"
server_key: "a0fbe17807098d9f6c52c9e623c1f2fe"
server_key_iv: "df88895d74cbb567e3527d20a739562a"
server_key_version: "encryptionkey@0"
I20260925 07:30:11.928206 13521 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790321392278483-9480-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790321392278483-9480-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790321392278483-9480-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:30:11.942672 13521 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:30:11.944554 13521 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:30:11.944710 13521 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:30:11.948612 13521 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.62:44971
I20260925 07:30:11.948637 13587 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.62:44971 every 8 connection(s)
I20260925 07:30:11.948909 13521 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
I20260925 07:30:11.949863 13588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:30:11.952064 13588 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef9eed146bc14a37a5428c146316db1d: Bootstrap starting.
I20260925 07:30:11.952661 13588 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ef9eed146bc14a37a5428c146316db1d: Neither blocks nor log segments found. Creating new log.
I20260925 07:30:11.952968 13588 log.cc:824] T 00000000000000000000000000000000 P ef9eed146bc14a37a5428c146316db1d: Log is configured to *not* fsync() on all Append() calls
I20260925 07:30:11.953629 13588 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef9eed146bc14a37a5428c146316db1d: No bootstrap required, opened a new log
I20260925 07:30:11.955222 13588 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef9eed146bc14a37a5428c146316db1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef9eed146bc14a37a5428c146316db1d" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 44971 } }
I20260925 07:30:11.955350 13588 raft_consensus.cc:374] T 00000000000000000000000000000000 P ef9eed146bc14a37a5428c146316db1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:30:11.955371 13588 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef9eed146bc14a37a5428c146316db1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef9eed146bc14a37a5428c146316db1d, State: Initialized, Role: FOLLOWER
I20260925 07:30:11.955523 13588 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef9eed146bc14a37a5428c146316db1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef9eed146bc14a37a5428c146316db1d" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 44971 } }
I20260925 07:30:11.955600 13588 raft_consensus.cc:388] T 00000000000000000000000000000000 P ef9eed146bc14a37a5428c146316db1d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 07:30:11.955621 13588 raft_consensus.cc:482] T 00000000000000000000000000000000 P ef9eed146bc14a37a5428c146316db1d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 07:30:11.955646 13588 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef9eed146bc14a37a5428c146316db1d [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:30:11.956462 13588 raft_consensus.cc:504] T 00000000000000000000000000000000 P ef9eed146bc14a37a5428c146316db1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef9eed146bc14a37a5428c146316db1d" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 44971 } }
I20260925 07:30:11.956547 13588 leader_election.cc:302] T 00000000000000000000000000000000 P ef9eed146bc14a37a5428c146316db1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef9eed146bc14a37a5428c146316db1d; no voters:
I20260925 07:30:11.956743 13588 leader_election.cc:288] T 00000000000000000000000000000000 P ef9eed146bc14a37a5428c146316db1d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 07:30:11.956785 13591 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ef9eed146bc14a37a5428c146316db1d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:30:11.956957 13591 raft_consensus.cc:686] T 00000000000000000000000000000000 P ef9eed146bc14a37a5428c146316db1d [term 1 LEADER]: Becoming Leader. State: Replica: ef9eed146bc14a37a5428c146316db1d, State: Running, Role: LEADER
I20260925 07:30:11.957070 13591 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef9eed146bc14a37a5428c146316db1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef9eed146bc14a37a5428c146316db1d" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 44971 } }
I20260925 07:30:11.957077 13588 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef9eed146bc14a37a5428c146316db1d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:30:11.957463 13593 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef9eed146bc14a37a5428c146316db1d [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef9eed146bc14a37a5428c146316db1d. Latest consensus state: current_term: 1 leader_uuid: "ef9eed146bc14a37a5428c146316db1d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef9eed146bc14a37a5428c146316db1d" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 44971 } } }
I20260925 07:30:11.957495 13592 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef9eed146bc14a37a5428c146316db1d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ef9eed146bc14a37a5428c146316db1d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef9eed146bc14a37a5428c146316db1d" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 44971 } } }
I20260925 07:30:11.957659 13593 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef9eed146bc14a37a5428c146316db1d [sys.catalog]: This master's current role is: LEADER
I20260925 07:30:11.957676 13592 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef9eed146bc14a37a5428c146316db1d [sys.catalog]: This master's current role is: LEADER
I20260925 07:30:11.958019 13600 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 07:30:11.958456 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 13521
I20260925 07:30:11.958561 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790321392278483-9480-0/minicluster-data/master-0/wal/instance
I20260925 07:30:11.958614 13600 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 07:30:11.958750 9480 external_mini_cluster.cc:1428] Setting key 8ad1cb522d23a7b54678e3cc09ebd8d4
I20260925 07:30:11.960265 13600 catalog_manager.cc:1347] Generated new cluster ID: 24e8de4a124b4a38b8d9303ffe26cc3b
I20260925 07:30:11.960318 13600 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 07:30:11 dist-test-slave-57fn krb5kdc[13508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321411, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.66.62@KRBTEST.COM
I20260925 07:30:11.968984 13600 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 07:30:11.969687 13600 catalog_manager.cc:1504] Loading token signing keys...
I20260925 07:30:11.971065 13610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:11.959186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38514 (local address 127.9.66.62:44971)
0925 07:30:11.959717 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:11.959725 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:11.959765 (+ 40us) server_negotiation.cc:407] Connection header received
0925 07:30:11.959895 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:11.959903 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:11.959958 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:11.960098 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:30:11.960882 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:11.962293 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:11.963369 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:11.963980 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:11.964070 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:11.967021 (+ 2951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:30:11.967025 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:30:11.967026 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:30:11.967038 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:30:11.969054 (+ 2016us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:11.969494 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:11.969497 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:11.969498 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:11.969628 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:11.969862 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:11.969863 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:11.969863 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:11.970103 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:30:11.970637 (+ 534us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:11.970701 (+ 64us) server_negotiation.cc:299] Negotiation successful
0925 07:30:11.970836 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":78,"threads_started":1}
I20260925 07:30:11.974118 13600 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ef9eed146bc14a37a5428c146316db1d: Generated new TSK 0
I20260925 07:30:11.974293 13600 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.66.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.66.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.66.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.1.keytab.
Entry for principal kudu/127.9.66.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.1.keytab.
I20260925 07:30:12.004885 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790321392278483-9480-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790321392278483-9480-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790321392278483-9480-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790321392278483-9480-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.66.1:0
--local_ip_for_outbound_sockets=127.9.66.1
--webserver_interface=127.9.66.1
--webserver_port=0
--tserver_master_addrs=127.9.66.62:44971
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.1.keytab
--principal=kudu/127.9.66.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:40037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:30:12.086506 13614 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:30:12.086726 13614 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:30:12.086750 13614 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:30:12.087903 13614 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:30:12.087944 13614 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:30:12.087971 13614 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.66.1
I20260925 07:30:12.089377 13614 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:40037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790321392278483-9480-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790321392278483-9480-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.1.keytab
--principal=kudu/127.9.66.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.66.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790321392278483-9480-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.66.1
--webserver_port=0
--tserver_master_addrs=127.9.66.62:44971
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13614
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.66.1
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790321392278483-9480-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:30:12.089633 13614 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:30:12.089828 13614 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:30:12.094720 13620 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:30:12.094748 13619 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:30:12.094724 13622 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:30:12.095005 13614 server_base.cc:1034] running on GCE node
I20260925 07:30:12.095213 13614 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:30:12.095410 13614 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:30:12.096648 13614 hybrid_clock.cc:648] HybridClock initialized: now 1790321412096629 us; error 34 us; skew 500 ppm
Sep 25 07:30:12 dist-test-slave-57fn krb5kdc[13508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321412, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:30:12.099236 13614 init.cc:377] Logged in from keytab as kudu/127.9.66.1@KRBTEST.COM (short username kudu)
I20260925 07:30:12.099628 13614 fs_manager.cc:359] Metadata directory not provided
I20260925 07:30:12.099676 13614 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:30:12.099824 13614 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:30:12.100813 13614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790321392278483-9480-0/minicluster-data/ts-0/data/instance:
uuid: "b29a2623d9444be8887cd3c562b58679"
format_stamp: "Formatted at 2026-09-25 07:30:12 on dist-test-slave-57fn"
server_key: "bc88efabefbbdeb3ed4d4c958061540c"
server_key_iv: "484dea2801039c9ecba14d9b49b65ee9"
server_key_version: "encryptionkey@0"
I20260925 07:30:12.101154 13614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790321392278483-9480-0/minicluster-data/ts-0/wal/instance:
uuid: "b29a2623d9444be8887cd3c562b58679"
format_stamp: "Formatted at 2026-09-25 07:30:12 on dist-test-slave-57fn"
server_key: "bc88efabefbbdeb3ed4d4c958061540c"
server_key_iv: "484dea2801039c9ecba14d9b49b65ee9"
server_key_version: "encryptionkey@0"
I20260925 07:30:12.102622 13614 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 07:30:12.103374 13628 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:12.103565 13614 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 07:30:12.103652 13614 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790321392278483-9480-0/minicluster-data/ts-0/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790321392278483-9480-0/minicluster-data/ts-0/wal
uuid: "b29a2623d9444be8887cd3c562b58679"
format_stamp: "Formatted at 2026-09-25 07:30:12 on dist-test-slave-57fn"
server_key: "bc88efabefbbdeb3ed4d4c958061540c"
server_key_iv: "484dea2801039c9ecba14d9b49b65ee9"
server_key_version: "encryptionkey@0"
I20260925 07:30:12.103724 13614 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790321392278483-9480-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790321392278483-9480-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790321392278483-9480-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:30:12.118666 13614 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:30:12.120893 13614 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:30:12.121073 13614 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:30:12.121310 13614 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:30:12.121718 13614 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:30:12.121770 13614 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:12.121826 13614 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:30:12.121855 13614 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:12.127628 13614 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.1:37801
I20260925 07:30:12.127665 13741 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.1:37801 every 8 connection(s)
I20260925 07:30:12.127907 13614 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790321392278483-9480-0/minicluster-data/ts-0/data/info.pb
I20260925 07:30:12.129675 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 13614
I20260925 07:30:12.129802 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790321392278483-9480-0/minicluster-data/ts-0/wal/instance
I20260925 07:30:12.129974 9480 external_mini_cluster.cc:1428] Setting key 96a2c581c591f499c76766bfaa4b7e26
Sep 25 07:30:12 dist-test-slave-57fn krb5kdc[13508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321412, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.1@KRBTEST.COM for kudu/127.9.66.62@KRBTEST.COM
I20260925 07:30:12.137656 13610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:12.128516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.1:54939 (local address 127.9.66.62:44971)
0925 07:30:12.128580 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:12.128582 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:12.128791 (+ 209us) server_negotiation.cc:407] Connection header received
0925 07:30:12.129273 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:12.129274 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:12.129287 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:12.129344 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:30:12.130441 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:12.131164 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:12.132374 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:12.132561 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:12.132616 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:12.134868 (+ 2252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:30:12.134870 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:30:12.134870 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:30:12.134877 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:30:12.136256 (+ 1379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:12.136652 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:12.136653 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:12.136654 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:12.136711 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:12.136960 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:12.136961 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:12.136961 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:12.137099 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:30:12.137170 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:12.137440 (+ 270us) server_negotiation.cc:299] Negotiation successful
0925 07:30:12.137550 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 07:30:12.137810 13744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:12.128494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:44971 (local address 127.9.66.1:54939)
0925 07:30:12.128664 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:12.128702 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 07:30:12.129185 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:30:12.129411 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:30:12.129414 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:30:12.129709 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:30:12.130358 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:12.130364 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:12.131252 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:12.131252 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:12.132299 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:12.132302 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:12.132643 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:12.132644 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:12.132778 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:12.133231 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:30:12.133241 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:30:12.134777 (+ 1536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:30:12.136348 (+ 1571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:30:12.136354 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:30:12.136356 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:30:12.136562 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:30:12.136783 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:30:12.136784 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:30:12.136784 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:30:12.136887 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:30:12.137174 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:30:12.137180 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:30:12.137308 (+ 128us) client_negotiation.cc:769] Sending connection context
0925 07:30:12.137421 (+ 113us) client_negotiation.cc:240] Negotiation successful
0925 07:30:12.137584 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":94,"threads_started":1}
WARNING: no policy specified for kudu/127.9.66.2@KRBTEST.COM; defaulting to no policy
I20260925 07:30:12.138418 13742 heartbeater.cc:347] Connected to a master server at 127.9.66.62:44971
I20260925 07:30:12.138517 13742 heartbeater.cc:464] Registering TS with master...
I20260925 07:30:12.138652 13742 heartbeater.cc:511] Master 127.9.66.62:44971 requested a full tablet report, sending...
I20260925 07:30:12.139130 13552 ts_manager.cc:194] Registered new tserver with Master: b29a2623d9444be8887cd3c562b58679 (127.9.66.1:37801)
I20260925 07:30:12.139705 13552 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.66.1@KRBTEST.COM'} at 127.9.66.1:54939
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.66.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.66.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.2.keytab.
Entry for principal kudu/127.9.66.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.2.keytab.
I20260925 07:30:12.154553 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790321392278483-9480-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790321392278483-9480-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790321392278483-9480-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790321392278483-9480-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.66.2:0
--local_ip_for_outbound_sockets=127.9.66.2
--webserver_interface=127.9.66.2
--webserver_port=0
--tserver_master_addrs=127.9.66.62:44971
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.2.keytab
--principal=kudu/127.9.66.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:40037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:30:12.233189 13747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:30:12.233433 13747 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:30:12.233472 13747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:30:12.234632 13747 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:30:12.234699 13747 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:30:12.234743 13747 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.66.2
I20260925 07:30:12.236212 13747 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:40037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790321392278483-9480-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790321392278483-9480-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.2.keytab
--principal=kudu/127.9.66.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.66.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790321392278483-9480-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.66.2
--webserver_port=0
--tserver_master_addrs=127.9.66.62:44971
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13747
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.66.2
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790321392278483-9480-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:30:12.236474 13747 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:30:12.236691 13747 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:30:12.241314 13755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:30:12.241321 13753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:30:12.241321 13752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:30:12.241568 13747 server_base.cc:1034] running on GCE node
I20260925 07:30:12.241770 13747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:30:12.241976 13747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:30:12.243121 13747 hybrid_clock.cc:648] HybridClock initialized: now 1790321412243115 us; error 28 us; skew 500 ppm
Sep 25 07:30:12 dist-test-slave-57fn krb5kdc[13508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321412, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:30:12.245543 13747 init.cc:377] Logged in from keytab as kudu/127.9.66.2@KRBTEST.COM (short username kudu)
I20260925 07:30:12.245884 13747 fs_manager.cc:359] Metadata directory not provided
I20260925 07:30:12.245927 13747 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:30:12.246057 13747 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:30:12.247022 13747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790321392278483-9480-0/minicluster-data/ts-1/data/instance:
uuid: "c818ebd3baba4ae78e868c1109c47884"
format_stamp: "Formatted at 2026-09-25 07:30:12 on dist-test-slave-57fn"
server_key: "4c6afa7fb79b4cb3fe560ad51b4a5b5f"
server_key_iv: "b1b6af3d9edffc24ed8ef78fcd439d5d"
server_key_version: "encryptionkey@0"
I20260925 07:30:12.247373 13747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790321392278483-9480-0/minicluster-data/ts-1/wal/instance:
uuid: "c818ebd3baba4ae78e868c1109c47884"
format_stamp: "Formatted at 2026-09-25 07:30:12 on dist-test-slave-57fn"
server_key: "4c6afa7fb79b4cb3fe560ad51b4a5b5f"
server_key_iv: "b1b6af3d9edffc24ed8ef78fcd439d5d"
server_key_version: "encryptionkey@0"
I20260925 07:30:12.248694 13747 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 07:30:12.249368 13761 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:12.249594 13747 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 07:30:12.249663 13747 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790321392278483-9480-0/minicluster-data/ts-1/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790321392278483-9480-0/minicluster-data/ts-1/wal
uuid: "c818ebd3baba4ae78e868c1109c47884"
format_stamp: "Formatted at 2026-09-25 07:30:12 on dist-test-slave-57fn"
server_key: "4c6afa7fb79b4cb3fe560ad51b4a5b5f"
server_key_iv: "b1b6af3d9edffc24ed8ef78fcd439d5d"
server_key_version: "encryptionkey@0"
I20260925 07:30:12.249764 13747 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790321392278483-9480-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790321392278483-9480-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790321392278483-9480-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:30:12.259495 13747 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:30:12.261750 13747 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:30:12.261905 13747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:30:12.262118 13747 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:30:12.262428 13747 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:30:12.262460 13747 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:12.262481 13747 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:30:12.262528 13747 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:12.268308 13747 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.2:43265
I20260925 07:30:12.268350 13874 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.2:43265 every 8 connection(s)
I20260925 07:30:12.268596 13747 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790321392278483-9480-0/minicluster-data/ts-1/data/info.pb
I20260925 07:30:12.268889 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 13747
I20260925 07:30:12.269006 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790321392278483-9480-0/minicluster-data/ts-1/wal/instance
I20260925 07:30:12.269212 9480 external_mini_cluster.cc:1428] Setting key 6640d0559db16699d47c20ff31607175
Sep 25 07:30:12 dist-test-slave-57fn krb5kdc[13508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321412, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.2@KRBTEST.COM for kudu/127.9.66.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.66.3@KRBTEST.COM; defaulting to no policy
I20260925 07:30:12.278326 13610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:12.269293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.2:41275 (local address 127.9.66.62:44971)
0925 07:30:12.269341 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:12.269343 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:12.269629 (+ 286us) server_negotiation.cc:407] Connection header received
0925 07:30:12.270233 (+ 604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:12.270234 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:12.270246 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:12.270318 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:30:12.271283 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:12.271778 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:12.272775 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:12.273013 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:12.273083 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:12.275486 (+ 2403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:30:12.275488 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:30:12.275488 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:30:12.275496 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:30:12.276872 (+ 1376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:12.277317 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:12.277318 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:12.277318 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:12.277378 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:12.277654 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:12.277655 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:12.277655 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:12.277772 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:30:12.277848 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:12.278134 (+ 286us) server_negotiation.cc:299] Negotiation successful
0925 07:30:12.278219 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 07:30:12.278455 13877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:12.269319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:44971 (local address 127.9.66.2:41275)
0925 07:30:12.269496 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:12.269559 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0925 07:30:12.270131 (+ 572us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:30:12.270385 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:30:12.270388 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:30:12.270633 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:30:12.271185 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:12.271189 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:12.271881 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:12.271882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:12.272662 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:12.272664 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:12.273100 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:12.273101 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:12.273218 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:12.273800 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:30:12.273811 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:30:12.275393 (+ 1582us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:30:12.276964 (+ 1571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:30:12.276970 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:30:12.276972 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:30:12.277245 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:30:12.277444 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:30:12.277445 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:30:12.277446 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:30:12.277581 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:30:12.277840 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:30:12.277845 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:30:12.277983 (+ 138us) client_negotiation.cc:769] Sending connection context
0925 07:30:12.278086 (+ 103us) client_negotiation.cc:240] Negotiation successful
0925 07:30:12.278239 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":76,"threads_started":1}
I20260925 07:30:12.278885 13875 heartbeater.cc:347] Connected to a master server at 127.9.66.62:44971
I20260925 07:30:12.279017 13875 heartbeater.cc:464] Registering TS with master...
I20260925 07:30:12.279218 13875 heartbeater.cc:511] Master 127.9.66.62:44971 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.66.3@KRBTEST.COM" created.
I20260925 07:30:12.279603 13552 ts_manager.cc:194] Registered new tserver with Master: c818ebd3baba4ae78e868c1109c47884 (127.9.66.2:43265)
I20260925 07:30:12.280076 13552 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.66.2@KRBTEST.COM'} at 127.9.66.2:41275
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.66.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.3.keytab.
Entry for principal kudu/127.9.66.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.3.keytab.
I20260925 07:30:12.295826 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790321392278483-9480-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790321392278483-9480-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790321392278483-9480-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790321392278483-9480-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.66.3:0
--local_ip_for_outbound_sockets=127.9.66.3
--webserver_interface=127.9.66.3
--webserver_port=0
--tserver_master_addrs=127.9.66.62:44971
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.3.keytab
--principal=kudu/127.9.66.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:40037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:30:12.375998 13880 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:30:12.376209 13880 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:30:12.376233 13880 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:30:12.377381 13880 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:30:12.377424 13880 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:30:12.377452 13880 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.66.3
I20260925 07:30:12.378952 13880 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:40037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790321392278483-9480-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790321392278483-9480-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.3.keytab
--principal=kudu/127.9.66.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.66.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790321392278483-9480-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.66.3
--webserver_port=0
--tserver_master_addrs=127.9.66.62:44971
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13880
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.66.3
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790321392278483-9480-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:30:12.379124 13880 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:30:12.379290 13880 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:30:12.384123 13886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:30:12.384160 13888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:30:12.384181 13885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:30:12.384351 13880 server_base.cc:1034] running on GCE node
I20260925 07:30:12.384538 13880 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:30:12.384802 13880 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:30:12.385967 13880 hybrid_clock.cc:648] HybridClock initialized: now 1790321412385951 us; error 32 us; skew 500 ppm
Sep 25 07:30:12 dist-test-slave-57fn krb5kdc[13508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321412, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:30:12.388454 13880 init.cc:377] Logged in from keytab as kudu/127.9.66.3@KRBTEST.COM (short username kudu)
I20260925 07:30:12.388784 13880 fs_manager.cc:359] Metadata directory not provided
I20260925 07:30:12.388850 13880 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:30:12.388998 13880 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:30:12.390029 13880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790321392278483-9480-0/minicluster-data/ts-2/data/instance:
uuid: "7088ea52fc2441f1b9663a9ee302c202"
format_stamp: "Formatted at 2026-09-25 07:30:12 on dist-test-slave-57fn"
server_key: "c96dae07424f6a5c427401f392924a06"
server_key_iv: "583bda4e5861ec266d55e9ef2f05c52d"
server_key_version: "encryptionkey@0"
I20260925 07:30:12.390375 13880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790321392278483-9480-0/minicluster-data/ts-2/wal/instance:
uuid: "7088ea52fc2441f1b9663a9ee302c202"
format_stamp: "Formatted at 2026-09-25 07:30:12 on dist-test-slave-57fn"
server_key: "c96dae07424f6a5c427401f392924a06"
server_key_iv: "583bda4e5861ec266d55e9ef2f05c52d"
server_key_version: "encryptionkey@0"
I20260925 07:30:12.391587 13880 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 07:30:12.392349 13894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:12.392551 13880 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:30:12.392647 13880 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790321392278483-9480-0/minicluster-data/ts-2/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790321392278483-9480-0/minicluster-data/ts-2/wal
uuid: "7088ea52fc2441f1b9663a9ee302c202"
format_stamp: "Formatted at 2026-09-25 07:30:12 on dist-test-slave-57fn"
server_key: "c96dae07424f6a5c427401f392924a06"
server_key_iv: "583bda4e5861ec266d55e9ef2f05c52d"
server_key_version: "encryptionkey@0"
I20260925 07:30:12.392724 13880 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790321392278483-9480-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790321392278483-9480-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790321392278483-9480-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:30:12.400669 13880 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:30:12.402443 13880 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:30:12.402616 13880 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:30:12.402834 13880 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:30:12.403123 13880 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:30:12.403156 13880 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:12.403177 13880 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:30:12.403221 13880 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:12.408875 13880 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.3:44143
I20260925 07:30:12.408942 14007 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.3:44143 every 8 connection(s)
I20260925 07:30:12.409185 13880 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790321392278483-9480-0/minicluster-data/ts-2/data/info.pb
I20260925 07:30:12.410041 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 13880
I20260925 07:30:12.410149 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790321392278483-9480-0/minicluster-data/ts-2/wal/instance
I20260925 07:30:12.410362 9480 external_mini_cluster.cc:1428] Setting key e347842d68654076685e2bd9b8b8602c
Sep 25 07:30:12 dist-test-slave-57fn krb5kdc[13508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321412, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.3@KRBTEST.COM for kudu/127.9.66.62@KRBTEST.COM
I20260925 07:30:12.418043 13610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:12.409890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.3:45067 (local address 127.9.66.62:44971)
0925 07:30:12.409931 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:12.409933 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:12.410188 (+ 255us) server_negotiation.cc:407] Connection header received
0925 07:30:12.410807 (+ 619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:12.410808 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:12.410820 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:12.410875 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:30:12.411777 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:12.412287 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:12.413259 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:12.413471 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:12.413595 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:12.415493 (+ 1898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:30:12.415495 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:30:12.415495 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:30:12.415502 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:30:12.416644 (+ 1142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:12.417066 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:12.417067 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:12.417068 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:12.417126 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:12.417374 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:12.417376 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:12.417376 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:12.417486 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:30:12.417609 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:12.417878 (+ 269us) server_negotiation.cc:299] Negotiation successful
0925 07:30:12.417954 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 07:30:12.418215 14010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:12.409896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:44971 (local address 127.9.66.3:45067)
0925 07:30:12.410058 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:12.410090 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0925 07:30:12.410699 (+ 609us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:30:12.410940 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:30:12.410944 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:30:12.411195 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:30:12.411657 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:12.411661 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:12.412380 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:12.412381 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:12.413161 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:12.413163 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:12.413603 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:12.413605 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:12.413737 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:12.414141 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:30:12.414148 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:30:12.415388 (+ 1240us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:30:12.416757 (+ 1369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:30:12.416762 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:30:12.416764 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:30:12.416975 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:30:12.417192 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:30:12.417193 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:30:12.417193 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:30:12.417303 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:30:12.417601 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:30:12.417607 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:30:12.417745 (+ 138us) client_negotiation.cc:769] Sending connection context
0925 07:30:12.417854 (+ 109us) client_negotiation.cc:240] Negotiation successful
0925 07:30:12.417995 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":94,"threads_started":1}
I20260925 07:30:12.418674 14008 heartbeater.cc:347] Connected to a master server at 127.9.66.62:44971
I20260925 07:30:12.418762 14008 heartbeater.cc:464] Registering TS with master...
I20260925 07:30:12.418926 14008 heartbeater.cc:511] Master 127.9.66.62:44971 requested a full tablet report, sending...
I20260925 07:30:12.419299 13552 ts_manager.cc:194] Registered new tserver with Master: 7088ea52fc2441f1b9663a9ee302c202 (127.9.66.3:44143)
I20260925 07:30:12.419668 13552 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.66.3@KRBTEST.COM'} at 127.9.66.3:45067
I20260925 07:30:12.422001 9480 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 07:30:12.428603 13610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:12.423266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38530 (local address 127.9.66.62:44971)
0925 07:30:12.423311 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:12.423313 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:12.423408 (+ 95us) server_negotiation.cc:407] Connection header received
0925 07:30:12.423458 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:12.423458 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:12.423465 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:12.423504 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:30:12.424009 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:12.424441 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:12.425123 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:12.425291 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:12.425334 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:12.426468 (+ 1134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:30:12.426469 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:30:12.426469 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:30:12.426474 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:30:12.427806 (+ 1332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:12.428081 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:12.428081 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:12.428082 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:12.428120 (+ 38us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:12.428227 (+ 107us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:12.428227 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:12.428227 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:12.428342 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:30:12.428409 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:12.428498 (+ 89us) server_negotiation.cc:299] Negotiation successful
0925 07:30:12.428548 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260925 07:30:12.430015 13552 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38530:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 07:30:12.431667 13552 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 07:30:12.437467 14017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:12.433670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.2:43265 (local address 127.0.0.1:34248)
0925 07:30:12.434022 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:12.434034 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 07:30:12.434104 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:30:12.434291 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:30:12.434294 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:30:12.434297 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:30:12.434361 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:12.434363 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:12.435374 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:12.435375 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:12.436274 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:12.436276 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:12.437208 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:12.437209 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:12.437260 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:12.437268 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 07:30:12.437350 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 07:30:12.437400 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":50,"threads_started":1}
I20260925 07:30:12.437479 14019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:12.433824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34248 (local address 127.9.66.2:43265)
0925 07:30:12.434004 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:12.434009 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:12.434108 (+ 99us) server_negotiation.cc:407] Connection header received
0925 07:30:12.434175 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:12.434177 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:12.434189 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:12.434253 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:30:12.434440 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:12.435274 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:12.436372 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:12.437065 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:12.437125 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:12.437183 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:12.437367 (+ 184us) server_negotiation.cc:299] Negotiation successful
0925 07:30:12.437406 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":78,"threads_started":1}
I20260925 07:30:12.438040 14016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:12.433691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.3:44143 (local address 127.0.0.1:42736)
0925 07:30:12.434055 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:12.434064 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 07:30:12.434104 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:30:12.434382 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:30:12.434383 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:30:12.434384 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:30:12.434452 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:12.434453 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:12.435998 (+ 1545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:12.435999 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:12.436861 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:12.436863 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:12.437823 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:12.437824 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:12.437878 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:12.437882 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 07:30:12.437941 (+ 59us) client_negotiation.cc:240] Negotiation successful
0925 07:30:12.437977 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"mutex_wait_us":2,"thread_start_us":86,"threads_started":1}
I20260925 07:30:12.438059 14020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:12.433879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42736 (local address 127.9.66.3:44143)
0925 07:30:12.434218 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:12.434222 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:12.434250 (+ 28us) server_negotiation.cc:407] Connection header received
0925 07:30:12.434286 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:12.434288 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:12.434302 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:12.434813 (+ 511us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:30:12.434885 (+ 72us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:12.435897 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:12.436945 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:12.437620 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:12.437801 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:12.437856 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:12.437955 (+ 99us) server_negotiation.cc:299] Negotiation successful
0925 07:30:12.437991 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":75,"threads_started":1}
I20260925 07:30:12.438316 13809 tablet_service.cc:1467] Processing CreateTablet for tablet 19989f8a5fd44318a0d4690ea6977616 (DEFAULT_TABLE table=test-table [id=938741bc65104f92b9d6ca5e5bb790e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:30:12.438701 14018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:12.433668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.1:37801 (local address 127.0.0.1:35522)
0925 07:30:12.433828 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:12.433865 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 07:30:12.433928 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:30:12.434678 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:30:12.434679 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:30:12.434680 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:30:12.434713 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:12.434714 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:12.435991 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:12.435991 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:12.437036 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:12.437038 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:12.438525 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:12.438526 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:12.438584 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:12.438588 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 07:30:12.438635 (+ 47us) client_negotiation.cc:240] Negotiation successful
0925 07:30:12.438660 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"mutex_wait_us":22,"thread_start_us":62,"threads_started":1}
I20260925 07:30:12.438694 13809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19989f8a5fd44318a0d4690ea6977616. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:30:12.438791 14021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:12.433909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35522 (local address 127.9.66.1:37801)
0925 07:30:12.434541 (+ 632us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:12.434545 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:12.434553 (+ 8us) server_negotiation.cc:407] Connection header received
0925 07:30:12.434591 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:12.434593 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:12.434608 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:12.434682 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:30:12.434810 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:12.435878 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:12.437126 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:12.438429 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:12.438522 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:12.438573 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:12.438657 (+ 84us) server_negotiation.cc:299] Negotiation successful
0925 07:30:12.438696 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":68,"threads_started":1}
I20260925 07:30:12.441051 14022 tablet_bootstrap.cc:492] T 19989f8a5fd44318a0d4690ea6977616 P c818ebd3baba4ae78e868c1109c47884: Bootstrap starting.
I20260925 07:30:12.441341 13942 tablet_service.cc:1467] Processing CreateTablet for tablet 19989f8a5fd44318a0d4690ea6977616 (DEFAULT_TABLE table=test-table [id=938741bc65104f92b9d6ca5e5bb790e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:30:12.441663 14022 tablet_bootstrap.cc:654] T 19989f8a5fd44318a0d4690ea6977616 P c818ebd3baba4ae78e868c1109c47884: Neither blocks nor log segments found. Creating new log.
I20260925 07:30:12.441660 13942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19989f8a5fd44318a0d4690ea6977616. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:30:12.442162 14022 log.cc:824] T 19989f8a5fd44318a0d4690ea6977616 P c818ebd3baba4ae78e868c1109c47884: Log is configured to *not* fsync() on all Append() calls
I20260925 07:30:12.442865 14022 tablet_bootstrap.cc:492] T 19989f8a5fd44318a0d4690ea6977616 P c818ebd3baba4ae78e868c1109c47884: No bootstrap required, opened a new log
I20260925 07:30:12.442945 14022 ts_tablet_manager.cc:1397] T 19989f8a5fd44318a0d4690ea6977616 P c818ebd3baba4ae78e868c1109c47884: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 07:30:12.444120 14024 tablet_bootstrap.cc:492] T 19989f8a5fd44318a0d4690ea6977616 P 7088ea52fc2441f1b9663a9ee302c202: Bootstrap starting.
I20260925 07:30:12.444273 14022 raft_consensus.cc:348] T 19989f8a5fd44318a0d4690ea6977616 P c818ebd3baba4ae78e868c1109c47884 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c818ebd3baba4ae78e868c1109c47884" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 43265 } } peers { permanent_uuid: "b29a2623d9444be8887cd3c562b58679" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 37801 } } peers { permanent_uuid: "7088ea52fc2441f1b9663a9ee302c202" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 44143 } }
I20260925 07:30:12.444446 14022 raft_consensus.cc:374] T 19989f8a5fd44318a0d4690ea6977616 P c818ebd3baba4ae78e868c1109c47884 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:30:12.444484 14022 raft_consensus.cc:729] T 19989f8a5fd44318a0d4690ea6977616 P c818ebd3baba4ae78e868c1109c47884 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c818ebd3baba4ae78e868c1109c47884, State: Initialized, Role: FOLLOWER
I20260925 07:30:12.444602 14022 consensus_queue.cc:260] T 19989f8a5fd44318a0d4690ea6977616 P c818ebd3baba4ae78e868c1109c47884 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c818ebd3baba4ae78e868c1109c47884" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 43265 } } peers { permanent_uuid: "b29a2623d9444be8887cd3c562b58679" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 37801 } } peers { permanent_uuid: "7088ea52fc2441f1b9663a9ee302c202" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 44143 } }
I20260925 07:30:12.444738 14024 tablet_bootstrap.cc:654] T 19989f8a5fd44318a0d4690ea6977616 P 7088ea52fc2441f1b9663a9ee302c202: Neither blocks nor log segments found. Creating new log.
I20260925 07:30:12.444797 13676 tablet_service.cc:1467] Processing CreateTablet for tablet 19989f8a5fd44318a0d4690ea6977616 (DEFAULT_TABLE table=test-table [id=938741bc65104f92b9d6ca5e5bb790e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:30:12.444900 13875 heartbeater.cc:503] Master 127.9.66.62:44971 was elected leader, sending a full tablet report...
I20260925 07:30:12.445072 13676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19989f8a5fd44318a0d4690ea6977616. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:30:12.445099 14022 ts_tablet_manager.cc:1428] T 19989f8a5fd44318a0d4690ea6977616 P c818ebd3baba4ae78e868c1109c47884: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260925 07:30:12.445355 14024 log.cc:824] T 19989f8a5fd44318a0d4690ea6977616 P 7088ea52fc2441f1b9663a9ee302c202: Log is configured to *not* fsync() on all Append() calls
I20260925 07:30:12.446151 14024 tablet_bootstrap.cc:492] T 19989f8a5fd44318a0d4690ea6977616 P 7088ea52fc2441f1b9663a9ee302c202: No bootstrap required, opened a new log
I20260925 07:30:12.446208 14024 ts_tablet_manager.cc:1397] T 19989f8a5fd44318a0d4690ea6977616 P 7088ea52fc2441f1b9663a9ee302c202: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260925 07:30:12.448283 14027 tablet_bootstrap.cc:492] T 19989f8a5fd44318a0d4690ea6977616 P b29a2623d9444be8887cd3c562b58679: Bootstrap starting.
I20260925 07:30:12.448196 14024 raft_consensus.cc:348] T 19989f8a5fd44318a0d4690ea6977616 P 7088ea52fc2441f1b9663a9ee302c202 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c818ebd3baba4ae78e868c1109c47884" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 43265 } } peers { permanent_uuid: "b29a2623d9444be8887cd3c562b58679" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 37801 } } peers { permanent_uuid: "7088ea52fc2441f1b9663a9ee302c202" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 44143 } }
I20260925 07:30:12.448349 14024 raft_consensus.cc:374] T 19989f8a5fd44318a0d4690ea6977616 P 7088ea52fc2441f1b9663a9ee302c202 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:30:12.448380 14024 raft_consensus.cc:729] T 19989f8a5fd44318a0d4690ea6977616 P 7088ea52fc2441f1b9663a9ee302c202 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7088ea52fc2441f1b9663a9ee302c202, State: Initialized, Role: FOLLOWER
I20260925 07:30:12.448499 14024 consensus_queue.cc:260] T 19989f8a5fd44318a0d4690ea6977616 P 7088ea52fc2441f1b9663a9ee302c202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c818ebd3baba4ae78e868c1109c47884" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 43265 } } peers { permanent_uuid: "b29a2623d9444be8887cd3c562b58679" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 37801 } } peers { permanent_uuid: "7088ea52fc2441f1b9663a9ee302c202" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 44143 } }
I20260925 07:30:12.448756 14024 ts_tablet_manager.cc:1428] T 19989f8a5fd44318a0d4690ea6977616 P 7088ea52fc2441f1b9663a9ee302c202: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 07:30:12.448784 14008 heartbeater.cc:503] Master 127.9.66.62:44971 was elected leader, sending a full tablet report...
I20260925 07:30:12.448923 14027 tablet_bootstrap.cc:654] T 19989f8a5fd44318a0d4690ea6977616 P b29a2623d9444be8887cd3c562b58679: Neither blocks nor log segments found. Creating new log.
I20260925 07:30:12.449565 14027 log.cc:824] T 19989f8a5fd44318a0d4690ea6977616 P b29a2623d9444be8887cd3c562b58679: Log is configured to *not* fsync() on all Append() calls
I20260925 07:30:12.450238 14027 tablet_bootstrap.cc:492] T 19989f8a5fd44318a0d4690ea6977616 P b29a2623d9444be8887cd3c562b58679: No bootstrap required, opened a new log
I20260925 07:30:12.450299 14027 ts_tablet_manager.cc:1397] T 19989f8a5fd44318a0d4690ea6977616 P b29a2623d9444be8887cd3c562b58679: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260925 07:30:12.451743 14027 raft_consensus.cc:348] T 19989f8a5fd44318a0d4690ea6977616 P b29a2623d9444be8887cd3c562b58679 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c818ebd3baba4ae78e868c1109c47884" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 43265 } } peers { permanent_uuid: "b29a2623d9444be8887cd3c562b58679" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 37801 } } peers { permanent_uuid: "7088ea52fc2441f1b9663a9ee302c202" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 44143 } }
I20260925 07:30:12.451861 14027 raft_consensus.cc:374] T 19989f8a5fd44318a0d4690ea6977616 P b29a2623d9444be8887cd3c562b58679 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:30:12.451884 14027 raft_consensus.cc:729] T 19989f8a5fd44318a0d4690ea6977616 P b29a2623d9444be8887cd3c562b58679 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b29a2623d9444be8887cd3c562b58679, State: Initialized, Role: FOLLOWER
I20260925 07:30:12.451956 14027 consensus_queue.cc:260] T 19989f8a5fd44318a0d4690ea6977616 P b29a2623d9444be8887cd3c562b58679 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c818ebd3baba4ae78e868c1109c47884" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 43265 } } peers { permanent_uuid: "b29a2623d9444be8887cd3c562b58679" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 37801 } } peers { permanent_uuid: "7088ea52fc2441f1b9663a9ee302c202" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 44143 } }
I20260925 07:30:12.452178 14027 ts_tablet_manager.cc:1428] T 19989f8a5fd44318a0d4690ea6977616 P b29a2623d9444be8887cd3c562b58679: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260925 07:30:12.452215 13742 heartbeater.cc:503] Master 127.9.66.62:44971 was elected leader, sending a full tablet report...
I20260925 07:30:12.516116 14028 raft_consensus.cc:482] T 19989f8a5fd44318a0d4690ea6977616 P 7088ea52fc2441f1b9663a9ee302c202 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 07:30:12.516235 14028 raft_consensus.cc:504] T 19989f8a5fd44318a0d4690ea6977616 P 7088ea52fc2441f1b9663a9ee302c202 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c818ebd3baba4ae78e868c1109c47884" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 43265 } } peers { permanent_uuid: "b29a2623d9444be8887cd3c562b58679" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 37801 } } peers { permanent_uuid: "7088ea52fc2441f1b9663a9ee302c202" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 44143 } }
I20260925 07:30:12.516609 14028 leader_election.cc:288] T 19989f8a5fd44318a0d4690ea6977616 P 7088ea52fc2441f1b9663a9ee302c202 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c818ebd3baba4ae78e868c1109c47884 (127.9.66.2:43265), b29a2623d9444be8887cd3c562b58679 (127.9.66.1:37801)
W20260925 07:30:12.519097 13876 tablet.cc:2369] T 19989f8a5fd44318a0d4690ea6977616 P c818ebd3baba4ae78e868c1109c47884: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 07:30:12.519548 14021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:12.516893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.3:54397 (local address 127.9.66.1:37801)
0925 07:30:12.516933 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:12.516936 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:12.517104 (+ 168us) server_negotiation.cc:407] Connection header received
0925 07:30:12.517135 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:12.517137 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:12.517148 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:12.517221 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:30:12.517386 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:12.517897 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:12.518832 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:12.519273 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:12.519343 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:12.519357 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:12.519469 (+ 112us) server_negotiation.cc:299] Negotiation successful
0925 07:30:12.519499 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 07:30:12.519593 14010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:12.516824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.2:43265 (local address 127.9.66.3:39643)
0925 07:30:12.516885 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:12.516896 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 07:30:12.516934 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:30:12.517091 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:30:12.517092 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:30:12.517093 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:30:12.517164 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:12.517166 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:12.517897 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:12.517898 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:12.518691 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:12.518693 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:12.519314 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:12.519315 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:12.519396 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:12.519400 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 07:30:12.519458 (+ 58us) client_negotiation.cc:240] Negotiation successful
0925 07:30:12.519508 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":6}
I20260925 07:30:12.519603 14031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:12.516824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.1:37801 (local address 127.9.66.3:54397)
0925 07:30:12.517024 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:12.517034 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 07:30:12.517088 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:30:12.517248 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:30:12.517249 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:30:12.517250 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:30:12.517323 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:12.517324 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:12.517992 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:12.517993 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:12.518722 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:12.518723 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:12.519337 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:12.519338 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:12.519404 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:12.519407 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 07:30:12.519460 (+ 53us) client_negotiation.cc:240] Negotiation successful
0925 07:30:12.519508 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"mutex_wait_us":41,"thread_start_us":102,"threads_started":1}
I20260925 07:30:12.519819 14019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:12.516892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.3:39643 (local address 127.9.66.2:43265)
0925 07:30:12.516933 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:12.516936 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:12.516945 (+ 9us) server_negotiation.cc:407] Connection header received
0925 07:30:12.516993 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:12.516994 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:12.517005 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:12.517055 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:30:12.517249 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:12.517827 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:12.518796 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:12.519209 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:12.519273 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:12.519283 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:12.519694 (+ 411us) server_negotiation.cc:299] Negotiation successful
0925 07:30:12.519740 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 07:30:12.520048 13696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19989f8a5fd44318a0d4690ea6977616" candidate_uuid: "7088ea52fc2441f1b9663a9ee302c202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b29a2623d9444be8887cd3c562b58679" is_pre_election: true
I20260925 07:30:12.520082 13829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19989f8a5fd44318a0d4690ea6977616" candidate_uuid: "7088ea52fc2441f1b9663a9ee302c202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c818ebd3baba4ae78e868c1109c47884" is_pre_election: true
I20260925 07:30:12.520195 13696 raft_consensus.cc:2454] T 19989f8a5fd44318a0d4690ea6977616 P b29a2623d9444be8887cd3c562b58679 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7088ea52fc2441f1b9663a9ee302c202 in term 0.
I20260925 07:30:12.520226 13829 raft_consensus.cc:2454] T 19989f8a5fd44318a0d4690ea6977616 P c818ebd3baba4ae78e868c1109c47884 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7088ea52fc2441f1b9663a9ee302c202 in term 0.
I20260925 07:30:12.520447 13895 leader_election.cc:302] T 19989f8a5fd44318a0d4690ea6977616 P 7088ea52fc2441f1b9663a9ee302c202 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7088ea52fc2441f1b9663a9ee302c202, b29a2623d9444be8887cd3c562b58679; no voters:
I20260925 07:30:12.520574 14028 raft_consensus.cc:2781] T 19989f8a5fd44318a0d4690ea6977616 P 7088ea52fc2441f1b9663a9ee302c202 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 07:30:12.520627 14028 raft_consensus.cc:482] T 19989f8a5fd44318a0d4690ea6977616 P 7088ea52fc2441f1b9663a9ee302c202 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 07:30:12.520649 14028 raft_consensus.cc:3037] T 19989f8a5fd44318a0d4690ea6977616 P 7088ea52fc2441f1b9663a9ee302c202 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:30:12.521297 14028 raft_consensus.cc:504] T 19989f8a5fd44318a0d4690ea6977616 P 7088ea52fc2441f1b9663a9ee302c202 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c818ebd3baba4ae78e868c1109c47884" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 43265 } } peers { permanent_uuid: "b29a2623d9444be8887cd3c562b58679" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 37801 } } peers { permanent_uuid: "7088ea52fc2441f1b9663a9ee302c202" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 44143 } }
I20260925 07:30:12.521463 14028 leader_election.cc:288] T 19989f8a5fd44318a0d4690ea6977616 P 7088ea52fc2441f1b9663a9ee302c202 [CANDIDATE]: Term 1 election: Requested vote from peers c818ebd3baba4ae78e868c1109c47884 (127.9.66.2:43265), b29a2623d9444be8887cd3c562b58679 (127.9.66.1:37801)
I20260925 07:30:12.521571 13829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19989f8a5fd44318a0d4690ea6977616" candidate_uuid: "7088ea52fc2441f1b9663a9ee302c202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c818ebd3baba4ae78e868c1109c47884"
I20260925 07:30:12.521646 13829 raft_consensus.cc:3037] T 19989f8a5fd44318a0d4690ea6977616 P c818ebd3baba4ae78e868c1109c47884 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:30:12.521673 13696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19989f8a5fd44318a0d4690ea6977616" candidate_uuid: "7088ea52fc2441f1b9663a9ee302c202" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b29a2623d9444be8887cd3c562b58679"
I20260925 07:30:12.521734 13696 raft_consensus.cc:3037] T 19989f8a5fd44318a0d4690ea6977616 P b29a2623d9444be8887cd3c562b58679 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:30:12.522528 13829 raft_consensus.cc:2454] T 19989f8a5fd44318a0d4690ea6977616 P c818ebd3baba4ae78e868c1109c47884 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7088ea52fc2441f1b9663a9ee302c202 in term 1.
I20260925 07:30:12.522641 13696 raft_consensus.cc:2454] T 19989f8a5fd44318a0d4690ea6977616 P b29a2623d9444be8887cd3c562b58679 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7088ea52fc2441f1b9663a9ee302c202 in term 1.
I20260925 07:30:12.522704 13897 leader_election.cc:302] T 19989f8a5fd44318a0d4690ea6977616 P 7088ea52fc2441f1b9663a9ee302c202 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7088ea52fc2441f1b9663a9ee302c202, c818ebd3baba4ae78e868c1109c47884; no voters:
I20260925 07:30:12.522819 14028 raft_consensus.cc:2781] T 19989f8a5fd44318a0d4690ea6977616 P 7088ea52fc2441f1b9663a9ee302c202 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:30:12.522976 14028 raft_consensus.cc:686] T 19989f8a5fd44318a0d4690ea6977616 P 7088ea52fc2441f1b9663a9ee302c202 [term 1 LEADER]: Becoming Leader. State: Replica: 7088ea52fc2441f1b9663a9ee302c202, State: Running, Role: LEADER
I20260925 07:30:12.523092 14028 consensus_queue.cc:237] T 19989f8a5fd44318a0d4690ea6977616 P 7088ea52fc2441f1b9663a9ee302c202 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c818ebd3baba4ae78e868c1109c47884" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 43265 } } peers { permanent_uuid: "b29a2623d9444be8887cd3c562b58679" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 37801 } } peers { permanent_uuid: "7088ea52fc2441f1b9663a9ee302c202" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 44143 } }
I20260925 07:30:12.523927 13552 catalog_manager.cc:5495] T 19989f8a5fd44318a0d4690ea6977616 P 7088ea52fc2441f1b9663a9ee302c202 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7088ea52fc2441f1b9663a9ee302c202 (127.9.66.3). New cstate: current_term: 1 leader_uuid: "7088ea52fc2441f1b9663a9ee302c202" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c818ebd3baba4ae78e868c1109c47884" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 43265 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b29a2623d9444be8887cd3c562b58679" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 37801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7088ea52fc2441f1b9663a9ee302c202" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 44143 } health_report { overall_health: HEALTHY } } }
I20260925 07:30:12.534796 14020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:12.532248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42742 (local address 127.9.66.3:44143)
0925 07:30:12.532292 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:12.532295 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:12.532313 (+ 18us) server_negotiation.cc:407] Connection header received
0925 07:30:12.532346 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:12.532347 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:12.532362 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:12.532431 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 07:30:12.532581 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:12.533039 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:12.533912 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:12.534129 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:12.534200 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:12.534421 (+ 221us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 07:30:12.534530 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 07:30:12.534616 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:12.534655 (+ 39us) server_negotiation.cc:299] Negotiation successful
0925 07:30:12.534673 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 07:30:12.536456 13696 raft_consensus.cc:1261] T 19989f8a5fd44318a0d4690ea6977616 P b29a2623d9444be8887cd3c562b58679 [term 1 FOLLOWER]: Refusing update from remote peer 7088ea52fc2441f1b9663a9ee302c202: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:30:12.536456 13829 raft_consensus.cc:1261] T 19989f8a5fd44318a0d4690ea6977616 P c818ebd3baba4ae78e868c1109c47884 [term 1 FOLLOWER]: Refusing update from remote peer 7088ea52fc2441f1b9663a9ee302c202: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:30:12.536839 14028 consensus_queue.cc:1035] T 19989f8a5fd44318a0d4690ea6977616 P 7088ea52fc2441f1b9663a9ee302c202 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c818ebd3baba4ae78e868c1109c47884" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 43265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:30:12.536990 14032 consensus_queue.cc:1035] T 19989f8a5fd44318a0d4690ea6977616 P 7088ea52fc2441f1b9663a9ee302c202 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b29a2623d9444be8887cd3c562b58679" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 37801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:30:12.539412 14035 mvcc.cc:205] Tried to move back new op lower bound from 7333156505746829312 to 7333156505695301632. Current Snapshot: MvccSnapshot[applied={T|T < 7333156505746829312}]
I20260925 07:30:12.539831 14036 mvcc.cc:205] Tried to move back new op lower bound from 7333156505746829312 to 7333156505695301632. Current Snapshot: MvccSnapshot[applied={T|T < 7333156505746829312}]
I20260925 07:30:12.540170 14037 mvcc.cc:205] Tried to move back new op lower bound from 7333156505746829312 to 7333156505695301632. Current Snapshot: MvccSnapshot[applied={T|T < 7333156505746829312}]
I20260925 07:30:12.541565 13551 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38530:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 07:30:12.541661 13551 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38530:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 07:30:12.542425 13551 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ef9eed146bc14a37a5428c146316db1d: Sending DeleteTablet for 3 replicas of tablet 19989f8a5fd44318a0d4690ea6977616
I20260925 07:30:12.542754 13942 tablet_service.cc:1514] Processing DeleteTablet for tablet 19989f8a5fd44318a0d4690ea6977616 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:30:12 UTC) from {username='kudu', principal='kudu/127.9.66.62@KRBTEST.COM'} at 127.0.0.1:42736
I20260925 07:30:12.542758 13676 tablet_service.cc:1514] Processing DeleteTablet for tablet 19989f8a5fd44318a0d4690ea6977616 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:30:12 UTC) from {username='kudu', principal='kudu/127.9.66.62@KRBTEST.COM'} at 127.0.0.1:35522
I20260925 07:30:12.542753 13809 tablet_service.cc:1514] Processing DeleteTablet for tablet 19989f8a5fd44318a0d4690ea6977616 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:30:12 UTC) from {username='kudu', principal='kudu/127.9.66.62@KRBTEST.COM'} at 127.0.0.1:34248
I20260925 07:30:12.542996 14047 tablet_replica.cc:330] T 19989f8a5fd44318a0d4690ea6977616 P c818ebd3baba4ae78e868c1109c47884: stopping tablet replica
I20260925 07:30:12.543064 14045 tablet_replica.cc:330] T 19989f8a5fd44318a0d4690ea6977616 P 7088ea52fc2441f1b9663a9ee302c202: stopping tablet replica
I20260925 07:30:12.543092 14047 raft_consensus.cc:2229] T 19989f8a5fd44318a0d4690ea6977616 P c818ebd3baba4ae78e868c1109c47884 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 07:30:12.543136 14045 raft_consensus.cc:2229] T 19989f8a5fd44318a0d4690ea6977616 P 7088ea52fc2441f1b9663a9ee302c202 [term 1 LEADER]: Raft consensus shutting down.
I20260925 07:30:12.543202 14047 raft_consensus.cc:2258] T 19989f8a5fd44318a0d4690ea6977616 P c818ebd3baba4ae78e868c1109c47884 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:30:12.543238 14045 raft_consensus.cc:2258] T 19989f8a5fd44318a0d4690ea6977616 P 7088ea52fc2441f1b9663a9ee302c202 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:30:12.543370 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 13614
I20260925 07:30:12.544379 14045 ts_tablet_manager.cc:1905] T 19989f8a5fd44318a0d4690ea6977616 P 7088ea52fc2441f1b9663a9ee302c202: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 07:30:12.545866 14047 ts_tablet_manager.cc:1905] T 19989f8a5fd44318a0d4690ea6977616 P c818ebd3baba4ae78e868c1109c47884: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 07:30:12.546085 14045 ts_tablet_manager.cc:1918] T 19989f8a5fd44318a0d4690ea6977616 P 7088ea52fc2441f1b9663a9ee302c202: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 07:30:12.546152 14045 log.cc:1196] T 19989f8a5fd44318a0d4690ea6977616 P 7088ea52fc2441f1b9663a9ee302c202: Deleting WAL directory at /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790321392278483-9480-0/minicluster-data/ts-2/wal/wals/19989f8a5fd44318a0d4690ea6977616
I20260925 07:30:12.546454 14045 ts_tablet_manager.cc:1939] T 19989f8a5fd44318a0d4690ea6977616 P 7088ea52fc2441f1b9663a9ee302c202: Deleting consensus metadata
I20260925 07:30:12.546864 13537 catalog_manager.cc:4841] TS 7088ea52fc2441f1b9663a9ee302c202 (127.9.66.3:44143): tablet 19989f8a5fd44318a0d4690ea6977616 (table test-table [id=938741bc65104f92b9d6ca5e5bb790e5]) successfully deleted
I20260925 07:30:12.547472 14047 ts_tablet_manager.cc:1918] T 19989f8a5fd44318a0d4690ea6977616 P c818ebd3baba4ae78e868c1109c47884: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 07:30:12.547528 14047 log.cc:1196] T 19989f8a5fd44318a0d4690ea6977616 P c818ebd3baba4ae78e868c1109c47884: Deleting WAL directory at /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790321392278483-9480-0/minicluster-data/ts-1/wal/wals/19989f8a5fd44318a0d4690ea6977616
I20260925 07:30:12.547780 14047 ts_tablet_manager.cc:1939] T 19989f8a5fd44318a0d4690ea6977616 P c818ebd3baba4ae78e868c1109c47884: Deleting consensus metadata
I20260925 07:30:12.548127 13538 catalog_manager.cc:4841] TS c818ebd3baba4ae78e868c1109c47884 (127.9.66.2:43265): tablet 19989f8a5fd44318a0d4690ea6977616 (table test-table [id=938741bc65104f92b9d6ca5e5bb790e5]) successfully deleted
W20260925 07:30:12.552433 13536 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.66.1:37801 (error 108)
I20260925 07:30:12.552624 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 13747
I20260925 07:30:12.553264 14018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:12.552769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.1:37801 (local address 127.0.0.1:35538)
0925 07:30:12.553169 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:12.553238 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.1:37801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382}
W20260925 07:30:12.553575 13536 catalog_manager.cc:4584] TS b29a2623d9444be8887cd3c562b58679 (127.9.66.1:37801): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 19989f8a5fd44318a0d4690ea6977616: Network error: Client connection negotiation failed: client connection to 127.9.66.1:37801: connect: Connection refused (error 111)
I20260925 07:30:12.557749 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 13880
I20260925 07:30:12.562400 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 13521
2026-09-25T07:30:12Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (1886 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-25T07:30:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T07:30:12Z Disabled control of system clock
I20260925 07:30:12.589056 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790321392278483-9480-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790321392278483-9480-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790321392278483-9480-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.66.62:42843
--webserver_interface=127.9.66.62
--webserver_port=0
--builtin_ntp_servers=127.9.66.20:39287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.66.62:42843
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 07:30:12.667963 14052 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:30:12.668166 14052 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:30:12.668190 14052 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:30:12.669229 14052 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:30:12.669266 14052 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:30:12.669281 14052 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:30:12.669294 14052 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:30:12.669307 14052 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:30:12.670586 14052 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:39287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790321392278483-9480-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790321392278483-9480-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.66.62:42843
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.66.62:42843
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.66.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14052
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790321392278483-9480-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:30:12.670774 14052 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:30:12.670997 14052 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:30:12.675686 14060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:30:12.675845 14058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:30:12.675916 14052 server_base.cc:1034] running on GCE node
W20260925 07:30:12.675719 14057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:30:12.676225 14052 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:30:12.676537 14052 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:30:12.677673 14052 hybrid_clock.cc:648] HybridClock initialized: now 1790321412677649 us; error 40 us; skew 500 ppm
I20260925 07:30:12.678856 14052 webserver.cc:460] Webserver started at http://127.9.66.62:39209/ using document root <none> and password file <none>
I20260925 07:30:12.679170 14052 fs_manager.cc:359] Metadata directory not provided
I20260925 07:30:12.679224 14052 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:30:12.679448 14052 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:30:12.680483 14052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790321392278483-9480-0/minicluster-data/master-0/data/instance:
uuid: "abaf25f2a7144a2eba89f9a4e9ad6342"
format_stamp: "Formatted at 2026-09-25 07:30:12 on dist-test-slave-57fn"
server_key: "efa82c5f0baab45dfcd63957c544caaa"
server_key_iv: "940a9511cf3eff3a437b0521a9c11b70"
server_key_version: "encryptionkey@0"
I20260925 07:30:12.680873 14052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790321392278483-9480-0/minicluster-data/master-0/wal/instance:
uuid: "abaf25f2a7144a2eba89f9a4e9ad6342"
format_stamp: "Formatted at 2026-09-25 07:30:12 on dist-test-slave-57fn"
server_key: "efa82c5f0baab45dfcd63957c544caaa"
server_key_iv: "940a9511cf3eff3a437b0521a9c11b70"
server_key_version: "encryptionkey@0"
I20260925 07:30:12.682310 14052 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 07:30:12.683076 14066 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:12.683386 14052 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 07:30:12.683477 14052 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790321392278483-9480-0/minicluster-data/master-0/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790321392278483-9480-0/minicluster-data/master-0/wal
uuid: "abaf25f2a7144a2eba89f9a4e9ad6342"
format_stamp: "Formatted at 2026-09-25 07:30:12 on dist-test-slave-57fn"
server_key: "efa82c5f0baab45dfcd63957c544caaa"
server_key_iv: "940a9511cf3eff3a437b0521a9c11b70"
server_key_version: "encryptionkey@0"
I20260925 07:30:12.683552 14052 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790321392278483-9480-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790321392278483-9480-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790321392278483-9480-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:30:12.686870 14052 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:30:12.687126 14052 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:30:12.687283 14052 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:30:12.691149 14052 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.62:42843
I20260925 07:30:12.691169 14118 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.62:42843 every 8 connection(s)
I20260925 07:30:12.691499 14052 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
I20260925 07:30:12.692117 14119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:30:12.692737 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 14052
I20260925 07:30:12.692867 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790321392278483-9480-0/minicluster-data/master-0/wal/instance
I20260925 07:30:12.693077 9480 external_mini_cluster.cc:1428] Setting key c582067521809e77d6fc137def6ee080
I20260925 07:30:12.695436 14119 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abaf25f2a7144a2eba89f9a4e9ad6342: Bootstrap starting.
I20260925 07:30:12.695616 14122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:12.693659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53420 (local address 127.9.66.62:42843)
0925 07:30:12.693861 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:12.693867 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:12.693890 (+ 23us) server_negotiation.cc:407] Connection header received
0925 07:30:12.694281 (+ 391us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:12.694287 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:12.694322 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:12.694436 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:30:12.695121 (+ 685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:30:12.695124 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:30:12.695124 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 07:30:12.695128 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:30:12.695169 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 07:30:12.695178 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:30:12.695236 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:12.695340 (+ 104us) server_negotiation.cc:299] Negotiation successful
0925 07:30:12.695414 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":95,"threads_started":1}
I20260925 07:30:12.696089 14119 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P abaf25f2a7144a2eba89f9a4e9ad6342: Neither blocks nor log segments found. Creating new log.
I20260925 07:30:12.696382 14119 log.cc:824] T 00000000000000000000000000000000 P abaf25f2a7144a2eba89f9a4e9ad6342: Log is configured to *not* fsync() on all Append() calls
I20260925 07:30:12.697057 14119 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abaf25f2a7144a2eba89f9a4e9ad6342: No bootstrap required, opened a new log
I20260925 07:30:12.698418 14119 raft_consensus.cc:348] T 00000000000000000000000000000000 P abaf25f2a7144a2eba89f9a4e9ad6342 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abaf25f2a7144a2eba89f9a4e9ad6342" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 42843 } }
I20260925 07:30:12.698565 14119 raft_consensus.cc:374] T 00000000000000000000000000000000 P abaf25f2a7144a2eba89f9a4e9ad6342 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:30:12.698638 14119 raft_consensus.cc:729] T 00000000000000000000000000000000 P abaf25f2a7144a2eba89f9a4e9ad6342 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abaf25f2a7144a2eba89f9a4e9ad6342, State: Initialized, Role: FOLLOWER
I20260925 07:30:12.698773 14119 consensus_queue.cc:260] T 00000000000000000000000000000000 P abaf25f2a7144a2eba89f9a4e9ad6342 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abaf25f2a7144a2eba89f9a4e9ad6342" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 42843 } }
I20260925 07:30:12.698872 14119 raft_consensus.cc:388] T 00000000000000000000000000000000 P abaf25f2a7144a2eba89f9a4e9ad6342 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 07:30:12.698913 14119 raft_consensus.cc:482] T 00000000000000000000000000000000 P abaf25f2a7144a2eba89f9a4e9ad6342 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 07:30:12.698962 14119 raft_consensus.cc:3037] T 00000000000000000000000000000000 P abaf25f2a7144a2eba89f9a4e9ad6342 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:30:12.699681 14119 raft_consensus.cc:504] T 00000000000000000000000000000000 P abaf25f2a7144a2eba89f9a4e9ad6342 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abaf25f2a7144a2eba89f9a4e9ad6342" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 42843 } }
I20260925 07:30:12.699826 14119 leader_election.cc:302] T 00000000000000000000000000000000 P abaf25f2a7144a2eba89f9a4e9ad6342 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: abaf25f2a7144a2eba89f9a4e9ad6342; no voters:
I20260925 07:30:12.699980 14119 leader_election.cc:288] T 00000000000000000000000000000000 P abaf25f2a7144a2eba89f9a4e9ad6342 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 07:30:12.700019 14124 raft_consensus.cc:2781] T 00000000000000000000000000000000 P abaf25f2a7144a2eba89f9a4e9ad6342 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:30:12.700201 14124 raft_consensus.cc:686] T 00000000000000000000000000000000 P abaf25f2a7144a2eba89f9a4e9ad6342 [term 1 LEADER]: Becoming Leader. State: Replica: abaf25f2a7144a2eba89f9a4e9ad6342, State: Running, Role: LEADER
I20260925 07:30:12.700212 14119 sys_catalog.cc:564] T 00000000000000000000000000000000 P abaf25f2a7144a2eba89f9a4e9ad6342 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:30:12.700309 14124 consensus_queue.cc:237] T 00000000000000000000000000000000 P abaf25f2a7144a2eba89f9a4e9ad6342 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abaf25f2a7144a2eba89f9a4e9ad6342" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 42843 } }
I20260925 07:30:12.700670 14125 sys_catalog.cc:455] T 00000000000000000000000000000000 P abaf25f2a7144a2eba89f9a4e9ad6342 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "abaf25f2a7144a2eba89f9a4e9ad6342" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abaf25f2a7144a2eba89f9a4e9ad6342" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 42843 } } }
I20260925 07:30:12.700860 14126 sys_catalog.cc:455] T 00000000000000000000000000000000 P abaf25f2a7144a2eba89f9a4e9ad6342 [sys.catalog]: SysCatalogTable state changed. Reason: New leader abaf25f2a7144a2eba89f9a4e9ad6342. Latest consensus state: current_term: 1 leader_uuid: "abaf25f2a7144a2eba89f9a4e9ad6342" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abaf25f2a7144a2eba89f9a4e9ad6342" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 42843 } } }
I20260925 07:30:12.700937 14125 sys_catalog.cc:458] T 00000000000000000000000000000000 P abaf25f2a7144a2eba89f9a4e9ad6342 [sys.catalog]: This master's current role is: LEADER
I20260925 07:30:12.700953 14126 sys_catalog.cc:458] T 00000000000000000000000000000000 P abaf25f2a7144a2eba89f9a4e9ad6342 [sys.catalog]: This master's current role is: LEADER
I20260925 07:30:12.701303 14135 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 07:30:12.701802 14135 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 07:30:12.703305 14135 catalog_manager.cc:1347] Generated new cluster ID: b75ebab5ba594066a72e176fef8dac98
I20260925 07:30:12.703358 14135 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 07:30:12.719365 14135 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 07:30:12.719554 14135 catalog_manager.cc:1504] Loading token signing keys...
I20260925 07:30:12.722160 14135 catalog_manager.cc:5868] T 00000000000000000000000000000000 P abaf25f2a7144a2eba89f9a4e9ad6342: Generated new TSK 0
I20260925 07:30:12.722334 14135 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 07:30:12.727713 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790321392278483-9480-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790321392278483-9480-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790321392278483-9480-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790321392278483-9480-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.66.1:0
--local_ip_for_outbound_sockets=127.9.66.1
--webserver_interface=127.9.66.1
--webserver_port=0
--tserver_master_addrs=127.9.66.62:42843
--builtin_ntp_servers=127.9.66.20:39287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 07:30:12.819482 14143 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:30:12.819708 14143 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:30:12.819743 14143 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:30:12.820895 14143 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:30:12.820962 14143 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:30:12.821005 14143 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.66.1
I20260925 07:30:12.822495 14143 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:39287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790321392278483-9480-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790321392278483-9480-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.66.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790321392278483-9480-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.66.1
--webserver_port=0
--tserver_master_addrs=127.9.66.62:42843
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14143
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.66.1
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790321392278483-9480-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:30:12.822747 14143 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:30:12.823030 14143 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:30:12.828702 14149 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:30:12.828679 14148 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:30:12.828672 14151 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:30:12.828923 14143 server_base.cc:1034] running on GCE node
I20260925 07:30:12.829125 14143 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:30:12.829368 14143 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:30:12.830549 14143 hybrid_clock.cc:648] HybridClock initialized: now 1790321412830541 us; error 32 us; skew 500 ppm
I20260925 07:30:12.831910 14143 webserver.cc:460] Webserver started at http://127.9.66.1:44569/ using document root <none> and password file <none>
I20260925 07:30:12.832149 14143 fs_manager.cc:359] Metadata directory not provided
I20260925 07:30:12.832201 14143 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:30:12.832320 14143 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:30:12.833609 14143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790321392278483-9480-0/minicluster-data/ts-0/data/instance:
uuid: "d424993f25f642f58a2bcbde720cd697"
format_stamp: "Formatted at 2026-09-25 07:30:12 on dist-test-slave-57fn"
server_key: "05066f18ea66d1479159ea3e9b6e0352"
server_key_iv: "89cd47444b049bb5062c1608d5edbbc5"
server_key_version: "encryptionkey@0"
I20260925 07:30:12.833998 14143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790321392278483-9480-0/minicluster-data/ts-0/wal/instance:
uuid: "d424993f25f642f58a2bcbde720cd697"
format_stamp: "Formatted at 2026-09-25 07:30:12 on dist-test-slave-57fn"
server_key: "05066f18ea66d1479159ea3e9b6e0352"
server_key_iv: "89cd47444b049bb5062c1608d5edbbc5"
server_key_version: "encryptionkey@0"
I20260925 07:30:12.835502 14143 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 07:30:12.836292 14157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:12.836477 14143 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 07:30:12.836561 14143 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790321392278483-9480-0/minicluster-data/ts-0/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790321392278483-9480-0/minicluster-data/ts-0/wal
uuid: "d424993f25f642f58a2bcbde720cd697"
format_stamp: "Formatted at 2026-09-25 07:30:12 on dist-test-slave-57fn"
server_key: "05066f18ea66d1479159ea3e9b6e0352"
server_key_iv: "89cd47444b049bb5062c1608d5edbbc5"
server_key_version: "encryptionkey@0"
I20260925 07:30:12.836625 14143 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790321392278483-9480-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790321392278483-9480-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790321392278483-9480-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:30:12.841075 14143 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:30:12.841323 14143 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:30:12.841435 14143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:30:12.841655 14143 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:30:12.842047 14143 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:30:12.842092 14143 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:12.842118 14143 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:30:12.842142 14143 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:12.848825 14143 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.1:44115
I20260925 07:30:12.848857 14270 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.1:44115 every 8 connection(s)
I20260925 07:30:12.849155 14143 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790321392278483-9480-0/minicluster-data/ts-0/data/info.pb
I20260925 07:30:12.851969 14122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:12.849897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.1:40019 (local address 127.9.66.62:42843)
0925 07:30:12.849950 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:12.849956 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:12.850158 (+ 202us) server_negotiation.cc:407] Connection header received
0925 07:30:12.850539 (+ 381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:12.850540 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:12.850546 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:12.850603 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:30:12.851738 (+ 1135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:30:12.851740 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:30:12.851740 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 07:30:12.851745 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:30:12.851760 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 07:30:12.851765 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:30:12.851832 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:12.851886 (+ 54us) server_negotiation.cc:299] Negotiation successful
0925 07:30:12.851905 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 07:30:12.852130 14273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:12.849875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:42843 (local address 127.9.66.1:40019)
0925 07:30:12.850038 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:12.850067 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0925 07:30:12.850458 (+ 391us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:30:12.850646 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:30:12.850649 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:30:12.850656 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:30:12.851581 (+ 925us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 07:30:12.851581 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:30:12.851607 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 07:30:12.851638 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:30:12.851830 (+ 192us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:30:12.851833 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:30:12.851835 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 07:30:12.851884 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 07:30:12.851927 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":86,"threads_started":1}
I20260925 07:30:12.852617 14271 heartbeater.cc:347] Connected to a master server at 127.9.66.62:42843
I20260925 07:30:12.852716 14271 heartbeater.cc:464] Registering TS with master...
I20260925 07:30:12.852866 14271 heartbeater.cc:511] Master 127.9.66.62:42843 requested a full tablet report, sending...
I20260925 07:30:12.852913 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 14143
I20260925 07:30:12.853011 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790321392278483-9480-0/minicluster-data/ts-0/wal/instance
I20260925 07:30:12.853183 9480 external_mini_cluster.cc:1428] Setting key 2f2c4532c04cfb6dbb73c014b1442978
I20260925 07:30:12.853394 14083 ts_manager.cc:194] Registered new tserver with Master: d424993f25f642f58a2bcbde720cd697 (127.9.66.1:44115)
I20260925 07:30:12.854321 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790321392278483-9480-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790321392278483-9480-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790321392278483-9480-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790321392278483-9480-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.66.2:0
--local_ip_for_outbound_sockets=127.9.66.2
--webserver_interface=127.9.66.2
--webserver_port=0
--tserver_master_addrs=127.9.66.62:42843
--builtin_ntp_servers=127.9.66.20:39287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 07:30:12.932739 14274 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:30:12.932962 14274 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:30:12.932998 14274 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:30:12.934171 14274 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:30:12.934243 14274 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:30:12.934309 14274 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.66.2
I20260925 07:30:12.935608 14274 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:39287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790321392278483-9480-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790321392278483-9480-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.66.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790321392278483-9480-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.66.2
--webserver_port=0
--tserver_master_addrs=127.9.66.62:42843
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14274
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.66.2
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790321392278483-9480-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:30:12.935871 14274 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:30:12.936138 14274 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:30:12.941094 14279 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:30:12.941177 14280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:30:12.941089 14282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:30:12.941370 14274 server_base.cc:1034] running on GCE node
I20260925 07:30:12.941640 14274 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:30:12.941931 14274 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:30:12.943097 14274 hybrid_clock.cc:648] HybridClock initialized: now 1790321412943088 us; error 27 us; skew 500 ppm
I20260925 07:30:12.944264 14274 webserver.cc:460] Webserver started at http://127.9.66.2:37519/ using document root <none> and password file <none>
I20260925 07:30:12.944486 14274 fs_manager.cc:359] Metadata directory not provided
I20260925 07:30:12.944530 14274 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:30:12.944686 14274 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:30:12.945739 14274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790321392278483-9480-0/minicluster-data/ts-1/data/instance:
uuid: "595d3119457d4ec2a78242a440400d4d"
format_stamp: "Formatted at 2026-09-25 07:30:12 on dist-test-slave-57fn"
server_key: "3cca92b53ced92db430523e96cc37852"
server_key_iv: "fe15440e8bfc7115e0028e4c9a31b457"
server_key_version: "encryptionkey@0"
I20260925 07:30:12.946074 14274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790321392278483-9480-0/minicluster-data/ts-1/wal/instance:
uuid: "595d3119457d4ec2a78242a440400d4d"
format_stamp: "Formatted at 2026-09-25 07:30:12 on dist-test-slave-57fn"
server_key: "3cca92b53ced92db430523e96cc37852"
server_key_iv: "fe15440e8bfc7115e0028e4c9a31b457"
server_key_version: "encryptionkey@0"
I20260925 07:30:12.947237 14274 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:30:12.947952 14288 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:12.948118 14274 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 07:30:12.948213 14274 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790321392278483-9480-0/minicluster-data/ts-1/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790321392278483-9480-0/minicluster-data/ts-1/wal
uuid: "595d3119457d4ec2a78242a440400d4d"
format_stamp: "Formatted at 2026-09-25 07:30:12 on dist-test-slave-57fn"
server_key: "3cca92b53ced92db430523e96cc37852"
server_key_iv: "fe15440e8bfc7115e0028e4c9a31b457"
server_key_version: "encryptionkey@0"
I20260925 07:30:12.948318 14274 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790321392278483-9480-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790321392278483-9480-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790321392278483-9480-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:30:12.951447 14274 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:30:12.951705 14274 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:30:12.951850 14274 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:30:12.952036 14274 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:30:12.952332 14274 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:30:12.952386 14274 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:12.952425 14274 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:30:12.952452 14274 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:12.958046 14274 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.2:45653
I20260925 07:30:12.958112 14401 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.2:45653 every 8 connection(s)
I20260925 07:30:12.958398 14274 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790321392278483-9480-0/minicluster-data/ts-1/data/info.pb
I20260925 07:30:12.958604 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 14274
I20260925 07:30:12.958704 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790321392278483-9480-0/minicluster-data/ts-1/wal/instance
I20260925 07:30:12.958894 9480 external_mini_cluster.cc:1428] Setting key 16e0b89f16c7b8f1692f09c346e95278
I20260925 07:30:12.959946 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790321392278483-9480-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790321392278483-9480-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790321392278483-9480-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790321392278483-9480-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.66.3:0
--local_ip_for_outbound_sockets=127.9.66.3
--webserver_interface=127.9.66.3
--webserver_port=0
--tserver_master_addrs=127.9.66.62:42843
--builtin_ntp_servers=127.9.66.20:39287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260925 07:30:12.961844 14122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:12.959193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.2:52809 (local address 127.9.66.62:42843)
0925 07:30:12.959242 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:12.959245 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:12.959470 (+ 225us) server_negotiation.cc:407] Connection header received
0925 07:30:12.960066 (+ 596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:12.960068 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:12.960075 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:12.960147 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:30:12.961436 (+ 1289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:30:12.961438 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:30:12.961438 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 07:30:12.961442 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:30:12.961460 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 07:30:12.961465 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:30:12.961621 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:12.961735 (+ 114us) server_negotiation.cc:299] Negotiation successful
0925 07:30:12.961751 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 07:30:12.961946 14404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:12.959175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:42843 (local address 127.9.66.2:52809)
0925 07:30:12.959344 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:12.959384 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 07:30:12.959961 (+ 577us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:30:12.960230 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:30:12.960235 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:30:12.960245 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:30:12.961258 (+ 1013us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 07:30:12.961259 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:30:12.961297 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 07:30:12.961335 (+ 38us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:30:12.961631 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:30:12.961635 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:30:12.961637 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 07:30:12.961716 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 07:30:12.961770 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":78,"threads_started":1}
I20260925 07:30:12.962368 14402 heartbeater.cc:347] Connected to a master server at 127.9.66.62:42843
I20260925 07:30:12.962450 14402 heartbeater.cc:464] Registering TS with master...
I20260925 07:30:12.962604 14402 heartbeater.cc:511] Master 127.9.66.62:42843 requested a full tablet report, sending...
I20260925 07:30:12.962958 14082 ts_manager.cc:194] Registered new tserver with Master: 595d3119457d4ec2a78242a440400d4d (127.9.66.2:45653)
W20260925 07:30:13.038554 14405 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:30:13.038748 14405 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:30:13.038769 14405 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:30:13.039853 14405 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:30:13.039882 14405 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:30:13.039906 14405 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.66.3
I20260925 07:30:13.041239 14405 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:39287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790321392278483-9480-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790321392278483-9480-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.66.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790321392278483-9480-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.66.3
--webserver_port=0
--tserver_master_addrs=127.9.66.62:42843
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14405
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.66.3
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790321392278483-9480-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:30:13.041412 14405 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:30:13.041719 14405 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:30:13.046404 14413 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:30:13.046464 14410 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:30:13.046423 14411 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:30:13.046747 14405 server_base.cc:1034] running on GCE node
I20260925 07:30:13.046962 14405 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:30:13.047173 14405 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:30:13.048349 14405 hybrid_clock.cc:648] HybridClock initialized: now 1790321413048360 us; error 51 us; skew 500 ppm
I20260925 07:30:13.049469 14405 webserver.cc:460] Webserver started at http://127.9.66.3:44069/ using document root <none> and password file <none>
I20260925 07:30:13.049707 14405 fs_manager.cc:359] Metadata directory not provided
I20260925 07:30:13.049769 14405 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:30:13.049886 14405 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:30:13.050923 14405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790321392278483-9480-0/minicluster-data/ts-2/data/instance:
uuid: "4966a3656ee24c3a88ac6669d4c992f6"
format_stamp: "Formatted at 2026-09-25 07:30:13 on dist-test-slave-57fn"
server_key: "8685212361c011df0b3e5221db298af8"
server_key_iv: "11ebcecc8989014e2ddece97821dd35d"
server_key_version: "encryptionkey@0"
I20260925 07:30:13.051255 14405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790321392278483-9480-0/minicluster-data/ts-2/wal/instance:
uuid: "4966a3656ee24c3a88ac6669d4c992f6"
format_stamp: "Formatted at 2026-09-25 07:30:13 on dist-test-slave-57fn"
server_key: "8685212361c011df0b3e5221db298af8"
server_key_iv: "11ebcecc8989014e2ddece97821dd35d"
server_key_version: "encryptionkey@0"
I20260925 07:30:13.052567 14405 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 07:30:13.053272 14419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:13.053440 14405 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 07:30:13.053545 14405 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790321392278483-9480-0/minicluster-data/ts-2/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790321392278483-9480-0/minicluster-data/ts-2/wal
uuid: "4966a3656ee24c3a88ac6669d4c992f6"
format_stamp: "Formatted at 2026-09-25 07:30:13 on dist-test-slave-57fn"
server_key: "8685212361c011df0b3e5221db298af8"
server_key_iv: "11ebcecc8989014e2ddece97821dd35d"
server_key_version: "encryptionkey@0"
I20260925 07:30:13.053642 14405 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790321392278483-9480-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790321392278483-9480-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790321392278483-9480-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:30:13.056788 14405 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:30:13.057029 14405 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:30:13.057163 14405 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:30:13.057389 14405 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:30:13.057703 14405 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:30:13.057756 14405 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:13.057801 14405 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:30:13.057827 14405 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:13.063580 14405 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.3:41887
I20260925 07:30:13.063618 14532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.3:41887 every 8 connection(s)
I20260925 07:30:13.063913 14405 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790321392278483-9480-0/minicluster-data/ts-2/data/info.pb
I20260925 07:30:13.065079 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 14405
I20260925 07:30:13.065186 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790321392278483-9480-0/minicluster-data/ts-2/wal/instance
I20260925 07:30:13.065371 9480 external_mini_cluster.cc:1428] Setting key acaf0b094bea3bf52114780bf103a0d2
I20260925 07:30:13.067058 14122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:13.064563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.3:53951 (local address 127.9.66.62:42843)
0925 07:30:13.064622 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:13.064623 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:13.064860 (+ 237us) server_negotiation.cc:407] Connection header received
0925 07:30:13.065607 (+ 747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:13.065609 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:13.065616 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:13.065674 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:30:13.066878 (+ 1204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:30:13.066879 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:30:13.066879 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 07:30:13.066883 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:30:13.066895 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 07:30:13.066899 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:30:13.066949 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:13.066997 (+ 48us) server_negotiation.cc:299] Negotiation successful
0925 07:30:13.067009 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 07:30:13.067270 14535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:13.064550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:42843 (local address 127.9.66.3:53951)
0925 07:30:13.064742 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:13.064779 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 07:30:13.065402 (+ 623us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:30:13.065758 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:30:13.065763 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:30:13.065773 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:30:13.066740 (+ 967us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 07:30:13.066741 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:30:13.066768 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 07:30:13.066798 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:30:13.066951 (+ 153us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:30:13.066953 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:30:13.066956 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 07:30:13.067004 (+ 48us) client_negotiation.cc:240] Negotiation successful
0925 07:30:13.067048 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":91,"threads_started":1}
I20260925 07:30:13.067664 14533 heartbeater.cc:347] Connected to a master server at 127.9.66.62:42843
I20260925 07:30:13.067755 14533 heartbeater.cc:464] Registering TS with master...
I20260925 07:30:13.067863 14533 heartbeater.cc:511] Master 127.9.66.62:42843 requested a full tablet report, sending...
I20260925 07:30:13.068192 14082 ts_manager.cc:194] Registered new tserver with Master: 4966a3656ee24c3a88ac6669d4c992f6 (127.9.66.3:41887)
I20260925 07:30:13.077196 9480 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 07:30:13.079735 14122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:13.078572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53436 (local address 127.9.66.62:42843)
0925 07:30:13.078658 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:13.078660 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:13.078757 (+ 97us) server_negotiation.cc:407] Connection header received
0925 07:30:13.078847 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:13.078848 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:13.078854 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:13.078905 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:30:13.079513 (+ 608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:30:13.079515 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:30:13.079515 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 07:30:13.079519 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:30:13.079533 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 07:30:13.079537 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:30:13.079589 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:13.079644 (+ 55us) server_negotiation.cc:299] Negotiation successful
0925 07:30:13.079659 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 07:30:13.081341 14082 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53436:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 07:30:13.082957 14082 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 07:30:13.087042 14544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:13.085288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.1:44115 (local address 127.0.0.1:37966)
0925 07:30:13.085491 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:13.085500 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 07:30:13.085582 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:30:13.085868 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:30:13.085875 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:30:13.085882 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:30:13.086405 (+ 523us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 07:30:13.086405 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:30:13.086421 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 07:30:13.086445 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:30:13.086881 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:30:13.086882 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:30:13.086882 (+ 0us) client_negotiation.cc:769] Sending connection context
0925 07:30:13.086934 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 07:30:13.086948 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":68,"threads_started":1}
I20260925 07:30:13.087069 14545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:13.085278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58062 (local address 127.9.66.3:41887)
0925 07:30:13.085715 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:13.085720 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:13.085738 (+ 18us) server_negotiation.cc:407] Connection header received
0925 07:30:13.085761 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:13.085763 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:13.085769 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:13.085849 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:30:13.086697 (+ 848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:30:13.086705 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:30:13.086705 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 07:30:13.086709 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:30:13.086743 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 07:30:13.086769 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:30:13.086824 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:13.086962 (+ 138us) server_negotiation.cc:299] Negotiation successful
0925 07:30:13.086981 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":90,"threads_started":1}
I20260925 07:30:13.087042 14541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:13.085060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.3:41887 (local address 127.0.0.1:58062)
0925 07:30:13.085217 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:13.085259 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 07:30:13.085315 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:30:13.086103 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:30:13.086104 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:30:13.086107 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:30:13.086569 (+ 462us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 07:30:13.086569 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:30:13.086575 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 07:30:13.086584 (+ 9us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:30:13.086849 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:30:13.086856 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:30:13.086859 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 07:30:13.086931 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 07:30:13.086951 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"mutex_wait_us":3,"thread_start_us":93,"threads_started":1}
I20260925 07:30:13.087285 14542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:13.085128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.2:45653 (local address 127.0.0.1:51294)
0925 07:30:13.085483 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:13.085494 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 07:30:13.085566 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:30:13.086512 (+ 946us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:30:13.086512 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:30:13.086515 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:30:13.086880 (+ 365us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 07:30:13.086881 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:30:13.086886 (+ 5us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 07:30:13.086899 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:30:13.087113 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:30:13.087114 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:30:13.087115 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 07:30:13.087172 (+ 57us) client_negotiation.cc:240] Negotiation successful
0925 07:30:13.087186 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":83,"threads_started":1}
I20260925 07:30:13.087311 14546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:13.085534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37966 (local address 127.9.66.1:44115)
0925 07:30:13.085715 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:13.085720 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:13.085731 (+ 11us) server_negotiation.cc:407] Connection header received
0925 07:30:13.085757 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:13.085759 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:13.085768 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:13.085849 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:30:13.086510 (+ 661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:30:13.086515 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:30:13.086516 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 07:30:13.086520 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:30:13.086560 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 07:30:13.086586 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:30:13.086681 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:13.087209 (+ 528us) server_negotiation.cc:299] Negotiation successful
0925 07:30:13.087226 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":94,"threads_started":1}
I20260925 07:30:13.087309 14543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:13.085238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51294 (local address 127.9.66.2:45653)
0925 07:30:13.086200 (+ 962us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:13.086210 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:13.086217 (+ 7us) server_negotiation.cc:407] Connection header received
0925 07:30:13.086252 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:13.086253 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:13.086259 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:13.086346 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:30:13.086984 (+ 638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:30:13.086989 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:30:13.086989 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 07:30:13.086992 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:30:13.087021 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 07:30:13.087037 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:30:13.087088 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:13.087208 (+ 120us) server_negotiation.cc:299] Negotiation successful
0925 07:30:13.087226 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":926,"thread_start_us":80,"threads_started":1}
I20260925 07:30:13.088037 14336 tablet_service.cc:1467] Processing CreateTablet for tablet e84e3cebef424f6895b30f68bf6ae697 (DEFAULT_TABLE table=test-table [id=e37f23be686e49348bc2ae05c6486e73]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:30:13.088217 14205 tablet_service.cc:1467] Processing CreateTablet for tablet e84e3cebef424f6895b30f68bf6ae697 (DEFAULT_TABLE table=test-table [id=e37f23be686e49348bc2ae05c6486e73]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:30:13.088481 14336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e84e3cebef424f6895b30f68bf6ae697. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:30:13.088574 14205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e84e3cebef424f6895b30f68bf6ae697. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:30:13.088542 14467 tablet_service.cc:1467] Processing CreateTablet for tablet e84e3cebef424f6895b30f68bf6ae697 (DEFAULT_TABLE table=test-table [id=e37f23be686e49348bc2ae05c6486e73]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:30:13.088845 14467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e84e3cebef424f6895b30f68bf6ae697. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:30:13.091300 14547 tablet_bootstrap.cc:492] T e84e3cebef424f6895b30f68bf6ae697 P d424993f25f642f58a2bcbde720cd697: Bootstrap starting.
I20260925 07:30:13.091890 14549 tablet_bootstrap.cc:492] T e84e3cebef424f6895b30f68bf6ae697 P 595d3119457d4ec2a78242a440400d4d: Bootstrap starting.
I20260925 07:30:13.092024 14547 tablet_bootstrap.cc:654] T e84e3cebef424f6895b30f68bf6ae697 P d424993f25f642f58a2bcbde720cd697: Neither blocks nor log segments found. Creating new log.
I20260925 07:30:13.092365 14547 log.cc:824] T e84e3cebef424f6895b30f68bf6ae697 P d424993f25f642f58a2bcbde720cd697: Log is configured to *not* fsync() on all Append() calls
I20260925 07:30:13.092525 14549 tablet_bootstrap.cc:654] T e84e3cebef424f6895b30f68bf6ae697 P 595d3119457d4ec2a78242a440400d4d: Neither blocks nor log segments found. Creating new log.
I20260925 07:30:13.092852 14549 log.cc:824] T e84e3cebef424f6895b30f68bf6ae697 P 595d3119457d4ec2a78242a440400d4d: Log is configured to *not* fsync() on all Append() calls
I20260925 07:30:13.093122 14547 tablet_bootstrap.cc:492] T e84e3cebef424f6895b30f68bf6ae697 P d424993f25f642f58a2bcbde720cd697: No bootstrap required, opened a new log
I20260925 07:30:13.093134 14548 tablet_bootstrap.cc:492] T e84e3cebef424f6895b30f68bf6ae697 P 4966a3656ee24c3a88ac6669d4c992f6: Bootstrap starting.
I20260925 07:30:13.093204 14547 ts_tablet_manager.cc:1397] T e84e3cebef424f6895b30f68bf6ae697 P d424993f25f642f58a2bcbde720cd697: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 07:30:13.093770 14548 tablet_bootstrap.cc:654] T e84e3cebef424f6895b30f68bf6ae697 P 4966a3656ee24c3a88ac6669d4c992f6: Neither blocks nor log segments found. Creating new log.
I20260925 07:30:13.094069 14548 log.cc:824] T e84e3cebef424f6895b30f68bf6ae697 P 4966a3656ee24c3a88ac6669d4c992f6: Log is configured to *not* fsync() on all Append() calls
I20260925 07:30:13.094584 14549 tablet_bootstrap.cc:492] T e84e3cebef424f6895b30f68bf6ae697 P 595d3119457d4ec2a78242a440400d4d: No bootstrap required, opened a new log
I20260925 07:30:13.094666 14549 ts_tablet_manager.cc:1397] T e84e3cebef424f6895b30f68bf6ae697 P 595d3119457d4ec2a78242a440400d4d: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260925 07:30:13.094763 14548 tablet_bootstrap.cc:492] T e84e3cebef424f6895b30f68bf6ae697 P 4966a3656ee24c3a88ac6669d4c992f6: No bootstrap required, opened a new log
I20260925 07:30:13.094841 14548 ts_tablet_manager.cc:1397] T e84e3cebef424f6895b30f68bf6ae697 P 4966a3656ee24c3a88ac6669d4c992f6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 07:30:13.095932 14547 raft_consensus.cc:348] T e84e3cebef424f6895b30f68bf6ae697 P d424993f25f642f58a2bcbde720cd697 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "595d3119457d4ec2a78242a440400d4d" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 45653 } } peers { permanent_uuid: "4966a3656ee24c3a88ac6669d4c992f6" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 41887 } } peers { permanent_uuid: "d424993f25f642f58a2bcbde720cd697" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 44115 } }
I20260925 07:30:13.096103 14547 raft_consensus.cc:374] T e84e3cebef424f6895b30f68bf6ae697 P d424993f25f642f58a2bcbde720cd697 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:30:13.096139 14547 raft_consensus.cc:729] T e84e3cebef424f6895b30f68bf6ae697 P d424993f25f642f58a2bcbde720cd697 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d424993f25f642f58a2bcbde720cd697, State: Initialized, Role: FOLLOWER
I20260925 07:30:13.096287 14547 consensus_queue.cc:260] T e84e3cebef424f6895b30f68bf6ae697 P d424993f25f642f58a2bcbde720cd697 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "595d3119457d4ec2a78242a440400d4d" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 45653 } } peers { permanent_uuid: "4966a3656ee24c3a88ac6669d4c992f6" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 41887 } } peers { permanent_uuid: "d424993f25f642f58a2bcbde720cd697" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 44115 } }
I20260925 07:30:13.096562 14547 ts_tablet_manager.cc:1428] T e84e3cebef424f6895b30f68bf6ae697 P d424993f25f642f58a2bcbde720cd697: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260925 07:30:13.096699 14271 heartbeater.cc:503] Master 127.9.66.62:42843 was elected leader, sending a full tablet report...
I20260925 07:30:13.096803 14548 raft_consensus.cc:348] T e84e3cebef424f6895b30f68bf6ae697 P 4966a3656ee24c3a88ac6669d4c992f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "595d3119457d4ec2a78242a440400d4d" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 45653 } } peers { permanent_uuid: "4966a3656ee24c3a88ac6669d4c992f6" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 41887 } } peers { permanent_uuid: "d424993f25f642f58a2bcbde720cd697" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 44115 } }
I20260925 07:30:13.096966 14548 raft_consensus.cc:374] T e84e3cebef424f6895b30f68bf6ae697 P 4966a3656ee24c3a88ac6669d4c992f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:30:13.097008 14548 raft_consensus.cc:729] T e84e3cebef424f6895b30f68bf6ae697 P 4966a3656ee24c3a88ac6669d4c992f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4966a3656ee24c3a88ac6669d4c992f6, State: Initialized, Role: FOLLOWER
I20260925 07:30:13.097132 14548 consensus_queue.cc:260] T e84e3cebef424f6895b30f68bf6ae697 P 4966a3656ee24c3a88ac6669d4c992f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "595d3119457d4ec2a78242a440400d4d" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 45653 } } peers { permanent_uuid: "4966a3656ee24c3a88ac6669d4c992f6" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 41887 } } peers { permanent_uuid: "d424993f25f642f58a2bcbde720cd697" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 44115 } }
I20260925 07:30:13.097379 14548 ts_tablet_manager.cc:1428] T e84e3cebef424f6895b30f68bf6ae697 P 4966a3656ee24c3a88ac6669d4c992f6: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 07:30:13.097498 14533 heartbeater.cc:503] Master 127.9.66.62:42843 was elected leader, sending a full tablet report...
I20260925 07:30:13.097556 14549 raft_consensus.cc:348] T e84e3cebef424f6895b30f68bf6ae697 P 595d3119457d4ec2a78242a440400d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "595d3119457d4ec2a78242a440400d4d" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 45653 } } peers { permanent_uuid: "4966a3656ee24c3a88ac6669d4c992f6" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 41887 } } peers { permanent_uuid: "d424993f25f642f58a2bcbde720cd697" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 44115 } }
I20260925 07:30:13.097713 14549 raft_consensus.cc:374] T e84e3cebef424f6895b30f68bf6ae697 P 595d3119457d4ec2a78242a440400d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:30:13.097751 14549 raft_consensus.cc:729] T e84e3cebef424f6895b30f68bf6ae697 P 595d3119457d4ec2a78242a440400d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 595d3119457d4ec2a78242a440400d4d, State: Initialized, Role: FOLLOWER
I20260925 07:30:13.097859 14549 consensus_queue.cc:260] T e84e3cebef424f6895b30f68bf6ae697 P 595d3119457d4ec2a78242a440400d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "595d3119457d4ec2a78242a440400d4d" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 45653 } } peers { permanent_uuid: "4966a3656ee24c3a88ac6669d4c992f6" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 41887 } } peers { permanent_uuid: "d424993f25f642f58a2bcbde720cd697" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 44115 } }
I20260925 07:30:13.098115 14549 ts_tablet_manager.cc:1428] T e84e3cebef424f6895b30f68bf6ae697 P 595d3119457d4ec2a78242a440400d4d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 07:30:13.098129 14402 heartbeater.cc:503] Master 127.9.66.62:42843 was elected leader, sending a full tablet report...
W20260925 07:30:13.099722 14272 tablet.cc:2369] T e84e3cebef424f6895b30f68bf6ae697 P d424993f25f642f58a2bcbde720cd697: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 07:30:13.209105 14403 tablet.cc:2369] T e84e3cebef424f6895b30f68bf6ae697 P 595d3119457d4ec2a78242a440400d4d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 07:30:13.314436 14534 tablet.cc:2369] T e84e3cebef424f6895b30f68bf6ae697 P 4966a3656ee24c3a88ac6669d4c992f6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 07:30:13.424248 14553 raft_consensus.cc:482] T e84e3cebef424f6895b30f68bf6ae697 P d424993f25f642f58a2bcbde720cd697 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 07:30:13.424376 14553 raft_consensus.cc:504] T e84e3cebef424f6895b30f68bf6ae697 P d424993f25f642f58a2bcbde720cd697 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "595d3119457d4ec2a78242a440400d4d" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 45653 } } peers { permanent_uuid: "4966a3656ee24c3a88ac6669d4c992f6" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 41887 } } peers { permanent_uuid: "d424993f25f642f58a2bcbde720cd697" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 44115 } }
I20260925 07:30:13.424734 14553 leader_election.cc:288] T e84e3cebef424f6895b30f68bf6ae697 P d424993f25f642f58a2bcbde720cd697 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 595d3119457d4ec2a78242a440400d4d (127.9.66.2:45653), 4966a3656ee24c3a88ac6669d4c992f6 (127.9.66.3:41887)
I20260925 07:30:13.426245 14557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:13.424931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.3:41887 (local address 127.9.66.1:33815)
0925 07:30:13.425074 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:13.425087 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 07:30:13.425193 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:30:13.425378 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:30:13.425379 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:30:13.425382 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:30:13.425861 (+ 479us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 07:30:13.425862 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:30:13.425868 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 07:30:13.425879 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:30:13.426054 (+ 175us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:30:13.426054 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:30:13.426055 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 07:30:13.426102 (+ 47us) client_negotiation.cc:240] Negotiation successful
0925 07:30:13.426115 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":70,"threads_started":1}
I20260925 07:30:13.426308 14545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:13.425020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.1:33815 (local address 127.9.66.3:41887)
0925 07:30:13.425070 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:13.425072 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:13.425169 (+ 97us) server_negotiation.cc:407] Connection header received
0925 07:30:13.425284 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:13.425286 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:13.425291 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:13.425341 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:30:13.425965 (+ 624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:30:13.425967 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:30:13.425967 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 07:30:13.425972 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:30:13.425986 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 07:30:13.425990 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:30:13.426048 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:13.426218 (+ 170us) server_negotiation.cc:299] Negotiation successful
0925 07:30:13.426233 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 07:30:13.426568 14543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:13.425056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.1:37775 (local address 127.9.66.2:45653)
0925 07:30:13.425098 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:13.425100 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:13.425206 (+ 106us) server_negotiation.cc:407] Connection header received
0925 07:30:13.425297 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:13.425298 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:13.425302 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:13.425380 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:30:13.426113 (+ 733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:30:13.426114 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:30:13.426114 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 07:30:13.426119 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:30:13.426132 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 07:30:13.426137 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:30:13.426193 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:13.426481 (+ 288us) server_negotiation.cc:299] Negotiation successful
0925 07:30:13.426498 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 07:30:13.426599 14487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e84e3cebef424f6895b30f68bf6ae697" candidate_uuid: "d424993f25f642f58a2bcbde720cd697" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4966a3656ee24c3a88ac6669d4c992f6" is_pre_election: true
I20260925 07:30:13.426740 14487 raft_consensus.cc:2454] T e84e3cebef424f6895b30f68bf6ae697 P 4966a3656ee24c3a88ac6669d4c992f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d424993f25f642f58a2bcbde720cd697 in term 0.
I20260925 07:30:13.426796 14556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:13.424936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.2:45653 (local address 127.9.66.1:37775)
0925 07:30:13.425083 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:13.425094 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 07:30:13.425189 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:30:13.425408 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:30:13.425409 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:30:13.425412 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:30:13.426022 (+ 610us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 07:30:13.426023 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:30:13.426030 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 07:30:13.426043 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:30:13.426219 (+ 176us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:30:13.426220 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:30:13.426220 (+ 0us) client_negotiation.cc:769] Sending connection context
0925 07:30:13.426706 (+ 486us) client_negotiation.cc:240] Negotiation successful
0925 07:30:13.426722 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":69,"threads_started":1}
I20260925 07:30:13.426981 14158 leader_election.cc:302] T e84e3cebef424f6895b30f68bf6ae697 P d424993f25f642f58a2bcbde720cd697 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4966a3656ee24c3a88ac6669d4c992f6, d424993f25f642f58a2bcbde720cd697; no voters:
I20260925 07:30:13.427117 14553 raft_consensus.cc:2781] T e84e3cebef424f6895b30f68bf6ae697 P d424993f25f642f58a2bcbde720cd697 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 07:30:13.427125 14356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e84e3cebef424f6895b30f68bf6ae697" candidate_uuid: "d424993f25f642f58a2bcbde720cd697" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "595d3119457d4ec2a78242a440400d4d" is_pre_election: true
I20260925 07:30:13.427166 14553 raft_consensus.cc:482] T e84e3cebef424f6895b30f68bf6ae697 P d424993f25f642f58a2bcbde720cd697 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 07:30:13.427217 14553 raft_consensus.cc:3037] T e84e3cebef424f6895b30f68bf6ae697 P d424993f25f642f58a2bcbde720cd697 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:30:13.427264 14356 raft_consensus.cc:2454] T e84e3cebef424f6895b30f68bf6ae697 P 595d3119457d4ec2a78242a440400d4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d424993f25f642f58a2bcbde720cd697 in term 0.
I20260925 07:30:13.427986 14553 raft_consensus.cc:504] T e84e3cebef424f6895b30f68bf6ae697 P d424993f25f642f58a2bcbde720cd697 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "595d3119457d4ec2a78242a440400d4d" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 45653 } } peers { permanent_uuid: "4966a3656ee24c3a88ac6669d4c992f6" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 41887 } } peers { permanent_uuid: "d424993f25f642f58a2bcbde720cd697" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 44115 } }
I20260925 07:30:13.428130 14553 leader_election.cc:288] T e84e3cebef424f6895b30f68bf6ae697 P d424993f25f642f58a2bcbde720cd697 [CANDIDATE]: Term 1 election: Requested vote from peers 595d3119457d4ec2a78242a440400d4d (127.9.66.2:45653), 4966a3656ee24c3a88ac6669d4c992f6 (127.9.66.3:41887)
I20260925 07:30:13.428261 14487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e84e3cebef424f6895b30f68bf6ae697" candidate_uuid: "d424993f25f642f58a2bcbde720cd697" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4966a3656ee24c3a88ac6669d4c992f6"
I20260925 07:30:13.428282 14356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e84e3cebef424f6895b30f68bf6ae697" candidate_uuid: "d424993f25f642f58a2bcbde720cd697" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "595d3119457d4ec2a78242a440400d4d"
I20260925 07:30:13.428355 14487 raft_consensus.cc:3037] T e84e3cebef424f6895b30f68bf6ae697 P 4966a3656ee24c3a88ac6669d4c992f6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:30:13.428352 14356 raft_consensus.cc:3037] T e84e3cebef424f6895b30f68bf6ae697 P 595d3119457d4ec2a78242a440400d4d [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:30:13.429041 14356 raft_consensus.cc:2454] T e84e3cebef424f6895b30f68bf6ae697 P 595d3119457d4ec2a78242a440400d4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d424993f25f642f58a2bcbde720cd697 in term 1.
I20260925 07:30:13.429104 14487 raft_consensus.cc:2454] T e84e3cebef424f6895b30f68bf6ae697 P 4966a3656ee24c3a88ac6669d4c992f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d424993f25f642f58a2bcbde720cd697 in term 1.
I20260925 07:30:13.429212 14160 leader_election.cc:302] T e84e3cebef424f6895b30f68bf6ae697 P d424993f25f642f58a2bcbde720cd697 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 595d3119457d4ec2a78242a440400d4d, d424993f25f642f58a2bcbde720cd697; no voters:
I20260925 07:30:13.429293 14553 raft_consensus.cc:2781] T e84e3cebef424f6895b30f68bf6ae697 P d424993f25f642f58a2bcbde720cd697 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:30:13.429433 14553 raft_consensus.cc:686] T e84e3cebef424f6895b30f68bf6ae697 P d424993f25f642f58a2bcbde720cd697 [term 1 LEADER]: Becoming Leader. State: Replica: d424993f25f642f58a2bcbde720cd697, State: Running, Role: LEADER
I20260925 07:30:13.429538 14553 consensus_queue.cc:237] T e84e3cebef424f6895b30f68bf6ae697 P d424993f25f642f58a2bcbde720cd697 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "595d3119457d4ec2a78242a440400d4d" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 45653 } } peers { permanent_uuid: "4966a3656ee24c3a88ac6669d4c992f6" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 41887 } } peers { permanent_uuid: "d424993f25f642f58a2bcbde720cd697" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 44115 } }
I20260925 07:30:13.430327 14081 catalog_manager.cc:5495] T e84e3cebef424f6895b30f68bf6ae697 P d424993f25f642f58a2bcbde720cd697 reported cstate change: term changed from 0 to 1, leader changed from <none> to d424993f25f642f58a2bcbde720cd697 (127.9.66.1). New cstate: current_term: 1 leader_uuid: "d424993f25f642f58a2bcbde720cd697" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "595d3119457d4ec2a78242a440400d4d" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 45653 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4966a3656ee24c3a88ac6669d4c992f6" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 41887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d424993f25f642f58a2bcbde720cd697" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 44115 } health_report { overall_health: HEALTHY } } }
I20260925 07:30:13.439429 14546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:13.438512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48830 (local address 127.9.66.1:44115)
0925 07:30:13.438558 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:13.438560 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:13.438569 (+ 9us) server_negotiation.cc:407] Connection header received
0925 07:30:13.438600 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:13.438601 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:13.438619 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:13.438713 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:30:13.439203 (+ 490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:30:13.439204 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:30:13.439204 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 07:30:13.439208 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:30:13.439220 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 07:30:13.439224 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:30:13.439266 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:13.439336 (+ 70us) server_negotiation.cc:299] Negotiation successful
0925 07:30:13.439349 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 07:30:13.441094 14487 raft_consensus.cc:1261] T e84e3cebef424f6895b30f68bf6ae697 P 4966a3656ee24c3a88ac6669d4c992f6 [term 1 FOLLOWER]: Refusing update from remote peer d424993f25f642f58a2bcbde720cd697: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:30:13.441126 14356 raft_consensus.cc:1261] T e84e3cebef424f6895b30f68bf6ae697 P 595d3119457d4ec2a78242a440400d4d [term 1 FOLLOWER]: Refusing update from remote peer d424993f25f642f58a2bcbde720cd697: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:30:13.441406 14553 consensus_queue.cc:1035] T e84e3cebef424f6895b30f68bf6ae697 P d424993f25f642f58a2bcbde720cd697 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4966a3656ee24c3a88ac6669d4c992f6" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 41887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:30:13.441584 14558 consensus_queue.cc:1035] T e84e3cebef424f6895b30f68bf6ae697 P d424993f25f642f58a2bcbde720cd697 [LEADER]: Connected to new peer: Peer: permanent_uuid: "595d3119457d4ec2a78242a440400d4d" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 45653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:30:13.444204 14562 mvcc.cc:205] Tried to move back new op lower bound from 7333156509452320768 to 7333156509408014336. Current Snapshot: MvccSnapshot[applied={T|T < 7333156509452320768}]
I20260925 07:30:13.444398 14563 mvcc.cc:205] Tried to move back new op lower bound from 7333156509452320768 to 7333156509408014336. Current Snapshot: MvccSnapshot[applied={T|T < 7333156509452320768}]
I20260925 07:30:13.446048 14082 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53436:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 07:30:13.446142 14082 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53436:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 07:30:13.446897 14082 catalog_manager.cc:5782] T 00000000000000000000000000000000 P abaf25f2a7144a2eba89f9a4e9ad6342: Sending DeleteTablet for 3 replicas of tablet e84e3cebef424f6895b30f68bf6ae697
I20260925 07:30:13.447221 14467 tablet_service.cc:1514] Processing DeleteTablet for tablet e84e3cebef424f6895b30f68bf6ae697 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:30:13 UTC) from {username='slave'} at 127.0.0.1:58062
I20260925 07:30:13.447235 14336 tablet_service.cc:1514] Processing DeleteTablet for tablet e84e3cebef424f6895b30f68bf6ae697 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:30:13 UTC) from {username='slave'} at 127.0.0.1:51294
I20260925 07:30:13.447355 14205 tablet_service.cc:1514] Processing DeleteTablet for tablet e84e3cebef424f6895b30f68bf6ae697 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:30:13 UTC) from {username='slave'} at 127.0.0.1:37966
I20260925 07:30:13.447557 14570 tablet_replica.cc:330] T e84e3cebef424f6895b30f68bf6ae697 P 4966a3656ee24c3a88ac6669d4c992f6: stopping tablet replica
I20260925 07:30:13.447662 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 14143
I20260925 07:30:13.447685 14570 raft_consensus.cc:2229] T e84e3cebef424f6895b30f68bf6ae697 P 4966a3656ee24c3a88ac6669d4c992f6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 07:30:13.447782 14570 raft_consensus.cc:2258] T e84e3cebef424f6895b30f68bf6ae697 P 4966a3656ee24c3a88ac6669d4c992f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:30:13.448259 14571 tablet_replica.cc:330] T e84e3cebef424f6895b30f68bf6ae697 P 595d3119457d4ec2a78242a440400d4d: stopping tablet replica
I20260925 07:30:13.448367 14571 raft_consensus.cc:2229] T e84e3cebef424f6895b30f68bf6ae697 P 595d3119457d4ec2a78242a440400d4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 07:30:13.448474 14571 raft_consensus.cc:2258] T e84e3cebef424f6895b30f68bf6ae697 P 595d3119457d4ec2a78242a440400d4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:30:13.448909 14571 ts_tablet_manager.cc:1905] T e84e3cebef424f6895b30f68bf6ae697 P 595d3119457d4ec2a78242a440400d4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 07:30:13.449331 14570 ts_tablet_manager.cc:1905] T e84e3cebef424f6895b30f68bf6ae697 P 4966a3656ee24c3a88ac6669d4c992f6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 07:30:13.450878 14570 ts_tablet_manager.cc:1918] T e84e3cebef424f6895b30f68bf6ae697 P 4966a3656ee24c3a88ac6669d4c992f6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 07:30:13.450927 14571 ts_tablet_manager.cc:1918] T e84e3cebef424f6895b30f68bf6ae697 P 595d3119457d4ec2a78242a440400d4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 07:30:13.450963 14570 log.cc:1196] T e84e3cebef424f6895b30f68bf6ae697 P 4966a3656ee24c3a88ac6669d4c992f6: Deleting WAL directory at /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790321392278483-9480-0/minicluster-data/ts-2/wal/wals/e84e3cebef424f6895b30f68bf6ae697
I20260925 07:30:13.451004 14571 log.cc:1196] T e84e3cebef424f6895b30f68bf6ae697 P 595d3119457d4ec2a78242a440400d4d: Deleting WAL directory at /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790321392278483-9480-0/minicluster-data/ts-1/wal/wals/e84e3cebef424f6895b30f68bf6ae697
I20260925 07:30:13.451217 14570 ts_tablet_manager.cc:1939] T e84e3cebef424f6895b30f68bf6ae697 P 4966a3656ee24c3a88ac6669d4c992f6: Deleting consensus metadata
I20260925 07:30:13.451296 14571 ts_tablet_manager.cc:1939] T e84e3cebef424f6895b30f68bf6ae697 P 595d3119457d4ec2a78242a440400d4d: Deleting consensus metadata
I20260925 07:30:13.451607 14067 catalog_manager.cc:4841] TS 4966a3656ee24c3a88ac6669d4c992f6 (127.9.66.3:41887): tablet e84e3cebef424f6895b30f68bf6ae697 (table test-table [id=e37f23be686e49348bc2ae05c6486e73]) successfully deleted
I20260925 07:30:13.451663 14069 catalog_manager.cc:4841] TS 595d3119457d4ec2a78242a440400d4d (127.9.66.2:45653): tablet e84e3cebef424f6895b30f68bf6ae697 (table test-table [id=e37f23be686e49348bc2ae05c6486e73]) successfully deleted
W20260925 07:30:13.455192 14067 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.66.1:44115 (error 108)
I20260925 07:30:13.455421 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 14274
I20260925 07:30:13.456184 14542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:13.455570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.1:44115 (local address 127.0.0.1:48840)
0925 07:30:13.456066 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:13.456113 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.1:44115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":448}
W20260925 07:30:13.456329 14067 catalog_manager.cc:4584] TS d424993f25f642f58a2bcbde720cd697 (127.9.66.1:44115): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e84e3cebef424f6895b30f68bf6ae697: Network error: Client connection negotiation failed: client connection to 127.9.66.1:44115: connect: Connection refused (error 111)
I20260925 07:30:13.460662 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 14405
I20260925 07:30:13.465133 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 14052
2026-09-25T07:30:13Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (901 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260925 07:30:13.484263 9480 mini_oidc.cc:152] Starting JWKS server
I20260925 07:30:13.485296 9480 webserver.cc:460] Webserver started at https://127.0.0.1:33529/ using document root <none> and password file <none>
I20260925 07:30:13.485404 9480 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 07:30:13.485723 9480 webserver.cc:460] Webserver started at http://127.0.0.1:43685/ using document root <none> and password file <none>
2026-09-25T07:30:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T07:30:13Z Disabled control of system clock
I20260925 07:30:13.493070 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790321392278483-9480-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790321392278483-9480-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790321392278483-9480-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.66.62:39197
--webserver_interface=127.9.66.62
--webserver_port=0
--builtin_ntp_servers=127.9.66.20:33455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.66.62:39197
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33529/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790321392278483-9480-0/testchainca.cert with env {}
W20260925 07:30:13.573166 14579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:30:13.573379 14579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:30:13.573401 14579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:30:13.574597 14579 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:30:13.574640 14579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:30:13.574653 14579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:30:13.574666 14579 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:30:13.574677 14579 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 07:30:13.574689 14579 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 07:30:13.574702 14579 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33529/jwks.json
I20260925 07:30:13.576130 14579 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:33455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790321392278483-9480-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790321392278483-9480-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.66.62:39197
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.66.62:39197
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33529/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.66.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790321392278483-9480-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790321392278483-9480-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:30:13.576323 14579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:30:13.576480 14579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:30:13.581328 14587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:30:13.581390 14585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:30:13.581329 14584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:30:13.581697 14579 server_base.cc:1034] running on GCE node
I20260925 07:30:13.581970 14579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:30:13.582295 14579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:30:13.583494 14579 hybrid_clock.cc:648] HybridClock initialized: now 1790321413583487 us; error 40 us; skew 500 ppm
I20260925 07:30:13.584812 14579 webserver.cc:460] Webserver started at http://127.9.66.62:45157/ using document root <none> and password file <none>
I20260925 07:30:13.585053 14579 fs_manager.cc:359] Metadata directory not provided
I20260925 07:30:13.585124 14579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:30:13.585286 14579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:30:13.586436 14579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790321392278483-9480-0/minicluster-data/master-0/data/instance:
uuid: "8d0ca34e06c54f2081c6d0f8b6702fd2"
format_stamp: "Formatted at 2026-09-25 07:30:13 on dist-test-slave-57fn"
server_key: "60f0a3b68a62923fc96bf4af7982043d"
server_key_iv: "c835590a076cc28b163d197bfbc09d44"
server_key_version: "encryptionkey@0"
I20260925 07:30:13.586813 14579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790321392278483-9480-0/minicluster-data/master-0/wal/instance:
uuid: "8d0ca34e06c54f2081c6d0f8b6702fd2"
format_stamp: "Formatted at 2026-09-25 07:30:13 on dist-test-slave-57fn"
server_key: "60f0a3b68a62923fc96bf4af7982043d"
server_key_iv: "c835590a076cc28b163d197bfbc09d44"
server_key_version: "encryptionkey@0"
I20260925 07:30:13.588192 14579 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 07:30:13.589049 14593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:13.589262 14579 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:30:13.589366 14579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790321392278483-9480-0/minicluster-data/master-0/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790321392278483-9480-0/minicluster-data/master-0/wal
uuid: "8d0ca34e06c54f2081c6d0f8b6702fd2"
format_stamp: "Formatted at 2026-09-25 07:30:13 on dist-test-slave-57fn"
server_key: "60f0a3b68a62923fc96bf4af7982043d"
server_key_iv: "c835590a076cc28b163d197bfbc09d44"
server_key_version: "encryptionkey@0"
I20260925 07:30:13.589460 14579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790321392278483-9480-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790321392278483-9480-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790321392278483-9480-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:30:13.603202 14579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:30:13.603520 14579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:30:13.603686 14579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:30:13.607946 14579 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.62:39197
I20260925 07:30:13.607978 14645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.62:39197 every 8 connection(s)
I20260925 07:30:13.608349 14579 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
I20260925 07:30:13.609413 14646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:30:13.611651 14646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d0ca34e06c54f2081c6d0f8b6702fd2: Bootstrap starting.
I20260925 07:30:13.612177 14646 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8d0ca34e06c54f2081c6d0f8b6702fd2: Neither blocks nor log segments found. Creating new log.
I20260925 07:30:13.612406 14646 log.cc:824] T 00000000000000000000000000000000 P 8d0ca34e06c54f2081c6d0f8b6702fd2: Log is configured to *not* fsync() on all Append() calls
I20260925 07:30:13.613073 14646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d0ca34e06c54f2081c6d0f8b6702fd2: No bootstrap required, opened a new log
I20260925 07:30:13.614590 14646 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d0ca34e06c54f2081c6d0f8b6702fd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d0ca34e06c54f2081c6d0f8b6702fd2" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 39197 } }
I20260925 07:30:13.614719 14646 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8d0ca34e06c54f2081c6d0f8b6702fd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:30:13.614740 14646 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d0ca34e06c54f2081c6d0f8b6702fd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d0ca34e06c54f2081c6d0f8b6702fd2, State: Initialized, Role: FOLLOWER
I20260925 07:30:13.614886 14646 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d0ca34e06c54f2081c6d0f8b6702fd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d0ca34e06c54f2081c6d0f8b6702fd2" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 39197 } }
I20260925 07:30:13.614966 14646 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8d0ca34e06c54f2081c6d0f8b6702fd2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 07:30:13.614989 14646 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d0ca34e06c54f2081c6d0f8b6702fd2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 07:30:13.615015 14646 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d0ca34e06c54f2081c6d0f8b6702fd2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:30:13.615473 14646 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d0ca34e06c54f2081c6d0f8b6702fd2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d0ca34e06c54f2081c6d0f8b6702fd2" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 39197 } }
I20260925 07:30:13.615551 14646 leader_election.cc:302] T 00000000000000000000000000000000 P 8d0ca34e06c54f2081c6d0f8b6702fd2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d0ca34e06c54f2081c6d0f8b6702fd2; no voters:
I20260925 07:30:13.615672 14646 leader_election.cc:288] T 00000000000000000000000000000000 P 8d0ca34e06c54f2081c6d0f8b6702fd2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 07:30:13.615859 14649 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d0ca34e06c54f2081c6d0f8b6702fd2 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:30:13.615944 14646 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d0ca34e06c54f2081c6d0f8b6702fd2 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:30:13.616024 14649 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d0ca34e06c54f2081c6d0f8b6702fd2 [term 1 LEADER]: Becoming Leader. State: Replica: 8d0ca34e06c54f2081c6d0f8b6702fd2, State: Running, Role: LEADER
I20260925 07:30:13.616117 14649 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d0ca34e06c54f2081c6d0f8b6702fd2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d0ca34e06c54f2081c6d0f8b6702fd2" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 39197 } }
I20260925 07:30:13.616541 14651 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d0ca34e06c54f2081c6d0f8b6702fd2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d0ca34e06c54f2081c6d0f8b6702fd2. Latest consensus state: current_term: 1 leader_uuid: "8d0ca34e06c54f2081c6d0f8b6702fd2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d0ca34e06c54f2081c6d0f8b6702fd2" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 39197 } } }
I20260925 07:30:13.616706 14650 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d0ca34e06c54f2081c6d0f8b6702fd2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8d0ca34e06c54f2081c6d0f8b6702fd2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d0ca34e06c54f2081c6d0f8b6702fd2" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 39197 } } }
I20260925 07:30:13.616783 14651 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d0ca34e06c54f2081c6d0f8b6702fd2 [sys.catalog]: This master's current role is: LEADER
I20260925 07:30:13.616801 14650 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d0ca34e06c54f2081c6d0f8b6702fd2 [sys.catalog]: This master's current role is: LEADER
I20260925 07:30:13.617100 14660 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 07:30:13.617339 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 14579
I20260925 07:30:13.617475 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790321392278483-9480-0/minicluster-data/master-0/wal/instance
I20260925 07:30:13.617717 9480 external_mini_cluster.cc:1428] Setting key 4ada899ca048b815e341de8553a82e17
I20260925 07:30:13.617828 14660 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 07:30:13.619378 14660 catalog_manager.cc:1347] Generated new cluster ID: 8954ff7c2d5c439a96e37c596f262b7c
I20260925 07:30:13.619428 14660 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 07:30:13.622822 14668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:13.618270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42546 (local address 127.9.66.62:39197)
0925 07:30:13.618533 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:13.618542 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:13.618583 (+ 41us) server_negotiation.cc:407] Connection header received
0925 07:30:13.618687 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:13.618693 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:13.618763 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:13.618886 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:30:13.619095 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:13.620065 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:13.620876 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:13.621300 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:13.621430 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:13.622143 (+ 713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:30:13.622146 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:30:13.622147 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 07:30:13.622320 (+ 173us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:30:13.622351 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 07:30:13.622361 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:30:13.622440 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:13.622511 (+ 71us) server_negotiation.cc:299] Negotiation successful
0925 07:30:13.622673 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":83,"threads_started":1}
I20260925 07:30:13.623643 9480 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 07:30:13.635954 14660 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 07:30:13.636420 14660 catalog_manager.cc:1504] Loading token signing keys...
I20260925 07:30:13.640832 14660 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8d0ca34e06c54f2081c6d0f8b6702fd2: Generated new TSK 0
I20260925 07:30:13.641000 14660 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260925 07:30:13.662436 14676 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260925 07:30:13.662532 14676 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 07:30:13.662882 14668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:13.647601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42562 (local address 127.9.66.62:39197)
0925 07:30:13.647654 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:13.647656 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:13.647771 (+ 115us) server_negotiation.cc:407] Connection header received
0925 07:30:13.647837 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:13.647838 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:13.647849 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:13.647923 (+ 74us) server_negotiation.cc:226] Negotiated authn=JWT
0925 07:30:13.648062 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:13.648844 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:13.649631 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:13.649936 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:13.650000 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:13.650176 (+ 176us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 07:30:13.662664 (+ 12488us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33529/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0925 07:30:13.662814 (+ 150us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42562: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33529/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":32}
W20260925 07:30:13.662940 14668 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42562: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33529/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260925 07:30:13.663182 14675 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:30:13.647527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:39197 (local address 127.0.0.1:42562)
0925 07:30:13.647708 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:13.647716 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 07:30:13.647773 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:30:13.647933 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:30:13.647934 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:30:13.647936 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0925 07:30:13.647998 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:13.647999 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:13.648916 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:13.648916 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:13.649555 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:13.649557 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:13.650002 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:13.650006 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:13.650051 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:13.650068 (+ 17us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 07:30:13.662789 (+ 12721us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33529/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0925 07:30:13.662874 (+ 85us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.9.66.62:39197: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33529/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":94,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.9.66.62:39197: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33529/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260925 07:30:13.663602 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 14579
2026-09-25T07:30:13Z chronyd exiting
I20260925 07:30:13.699266 9480 test_util.cc:175] -----------------------------------------------
I20260925 07:30:13.699365 9480 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790321392278483-9480-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (216 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260925 07:30:13.700209 9480 mini_oidc.cc:152] Starting JWKS server
I20260925 07:30:13.700881 9480 webserver.cc:460] Webserver started at https://127.0.0.1:39117/ using document root <none> and password file <none>
I20260925 07:30:13.700946 9480 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 07:30:13.701124 9480 webserver.cc:460] Webserver started at http://127.0.0.1:46739/ using document root <none> and password file <none>
2026-09-25T07:30:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T07:30:13Z Disabled control of system clock
I20260925 07:30:13.708401 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790321392278483-9480-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790321392278483-9480-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790321392278483-9480-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.66.62:33847
--webserver_interface=127.9.66.62
--webserver_port=0
--builtin_ntp_servers=127.9.66.20:41651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.66.62:33847
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39117/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790321392278483-9480-0/testchainca.cert with env {}
W20260925 07:30:13.786514 14682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:30:13.786778 14682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:30:13.786818 14682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:30:13.788007 14682 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:30:13.788084 14682 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:30:13.788107 14682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:30:13.788137 14682 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:30:13.788157 14682 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 07:30:13.788187 14682 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 07:30:13.788213 14682 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39117/jwks.json
I20260925 07:30:13.789788 14682 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:41651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790321392278483-9480-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790321392278483-9480-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.66.62:33847
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.66.62:33847
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39117/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.66.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790321392278483-9480-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14682
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790321392278483-9480-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:30:13.790046 14682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:30:13.790256 14682 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:30:13.795068 14687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:30:13.795100 14688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:30:13.795110 14690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:30:13.795159 14682 server_base.cc:1034] running on GCE node
I20260925 07:30:13.795526 14682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:30:13.795828 14682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:30:13.797010 14682 hybrid_clock.cc:648] HybridClock initialized: now 1790321413796992 us; error 50 us; skew 500 ppm
I20260925 07:30:13.798307 14682 webserver.cc:460] Webserver started at http://127.9.66.62:33527/ using document root <none> and password file <none>
I20260925 07:30:13.798568 14682 fs_manager.cc:359] Metadata directory not provided
I20260925 07:30:13.798635 14682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:30:13.798784 14682 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:30:13.799796 14682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790321392278483-9480-0/minicluster-data/master-0/data/instance:
uuid: "24c328a4e1c2419e9c2c7af8552c37e7"
format_stamp: "Formatted at 2026-09-25 07:30:13 on dist-test-slave-57fn"
server_key: "558483b56cb7588be3b506a3e18a5519"
server_key_iv: "4d7c21cb4eeda7d22bfac7bb92787a64"
server_key_version: "encryptionkey@0"
I20260925 07:30:13.800172 14682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790321392278483-9480-0/minicluster-data/master-0/wal/instance:
uuid: "24c328a4e1c2419e9c2c7af8552c37e7"
format_stamp: "Formatted at 2026-09-25 07:30:13 on dist-test-slave-57fn"
server_key: "558483b56cb7588be3b506a3e18a5519"
server_key_iv: "4d7c21cb4eeda7d22bfac7bb92787a64"
server_key_version: "encryptionkey@0"
I20260925 07:30:13.801595 14682 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 07:30:13.802373 14696 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:13.802556 14682 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:30:13.802645 14682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790321392278483-9480-0/minicluster-data/master-0/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790321392278483-9480-0/minicluster-data/master-0/wal
uuid: "24c328a4e1c2419e9c2c7af8552c37e7"
format_stamp: "Formatted at 2026-09-25 07:30:13 on dist-test-slave-57fn"
server_key: "558483b56cb7588be3b506a3e18a5519"
server_key_iv: "4d7c21cb4eeda7d22bfac7bb92787a64"
server_key_version: "encryptionkey@0"
I20260925 07:30:13.802722 14682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790321392278483-9480-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790321392278483-9480-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790321392278483-9480-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:30:13.834992 14682 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:30:13.835391 14682 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:30:13.835554 14682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:30:13.839619 14682 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.62:33847
I20260925 07:30:13.839738 14748 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.62:33847 every 8 connection(s)
I20260925 07:30:13.840031 14682 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
I20260925 07:30:13.840971 14749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:30:13.842886 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 14682
I20260925 07:30:13.843000 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790321392278483-9480-0/minicluster-data/master-0/wal/instance
I20260925 07:30:13.843191 9480 external_mini_cluster.cc:1428] Setting key 7faea99f469d72a1c99f2c89cba07f33
I20260925 07:30:13.843991 14749 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24c328a4e1c2419e9c2c7af8552c37e7: Bootstrap starting.
I20260925 07:30:13.844630 14749 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 24c328a4e1c2419e9c2c7af8552c37e7: Neither blocks nor log segments found. Creating new log.
I20260925 07:30:13.844959 14749 log.cc:824] T 00000000000000000000000000000000 P 24c328a4e1c2419e9c2c7af8552c37e7: Log is configured to *not* fsync() on all Append() calls
I20260925 07:30:13.845733 14749 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24c328a4e1c2419e9c2c7af8552c37e7: No bootstrap required, opened a new log
I20260925 07:30:13.848186 14749 raft_consensus.cc:348] T 00000000000000000000000000000000 P 24c328a4e1c2419e9c2c7af8552c37e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24c328a4e1c2419e9c2c7af8552c37e7" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 33847 } }
I20260925 07:30:13.848337 14749 raft_consensus.cc:374] T 00000000000000000000000000000000 P 24c328a4e1c2419e9c2c7af8552c37e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:30:13.848364 14749 raft_consensus.cc:729] T 00000000000000000000000000000000 P 24c328a4e1c2419e9c2c7af8552c37e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24c328a4e1c2419e9c2c7af8552c37e7, State: Initialized, Role: FOLLOWER
I20260925 07:30:13.848500 14749 consensus_queue.cc:260] T 00000000000000000000000000000000 P 24c328a4e1c2419e9c2c7af8552c37e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24c328a4e1c2419e9c2c7af8552c37e7" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 33847 } }
I20260925 07:30:13.848600 14749 raft_consensus.cc:388] T 00000000000000000000000000000000 P 24c328a4e1c2419e9c2c7af8552c37e7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 07:30:13.848642 14749 raft_consensus.cc:482] T 00000000000000000000000000000000 P 24c328a4e1c2419e9c2c7af8552c37e7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 07:30:13.848701 14749 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 24c328a4e1c2419e9c2c7af8552c37e7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:30:13.848985 14752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:13.843734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48700 (local address 127.9.66.62:33847)
0925 07:30:13.843928 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:13.843935 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:13.843959 (+ 24us) server_negotiation.cc:407] Connection header received
0925 07:30:13.844402 (+ 443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:13.844409 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:13.844468 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:13.844582 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:30:13.844846 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:13.845865 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:13.846952 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:13.847351 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:13.847453 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:13.848184 (+ 731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:30:13.848188 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:30:13.848188 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 07:30:13.848380 (+ 192us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:30:13.848424 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 07:30:13.848443 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:30:13.848561 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:13.848699 (+ 138us) server_negotiation.cc:299] Negotiation successful
0925 07:30:13.848785 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":90,"threads_started":1}
I20260925 07:30:13.849469 14749 raft_consensus.cc:504] T 00000000000000000000000000000000 P 24c328a4e1c2419e9c2c7af8552c37e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24c328a4e1c2419e9c2c7af8552c37e7" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 33847 } }
I20260925 07:30:13.849617 14749 leader_election.cc:302] T 00000000000000000000000000000000 P 24c328a4e1c2419e9c2c7af8552c37e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 24c328a4e1c2419e9c2c7af8552c37e7; no voters:
I20260925 07:30:13.849802 14749 leader_election.cc:288] T 00000000000000000000000000000000 P 24c328a4e1c2419e9c2c7af8552c37e7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 07:30:13.849826 14754 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 24c328a4e1c2419e9c2c7af8552c37e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:30:13.849988 14754 raft_consensus.cc:686] T 00000000000000000000000000000000 P 24c328a4e1c2419e9c2c7af8552c37e7 [term 1 LEADER]: Becoming Leader. State: Replica: 24c328a4e1c2419e9c2c7af8552c37e7, State: Running, Role: LEADER
I20260925 07:30:13.850040 14749 sys_catalog.cc:564] T 00000000000000000000000000000000 P 24c328a4e1c2419e9c2c7af8552c37e7 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:30:13.850116 14754 consensus_queue.cc:237] T 00000000000000000000000000000000 P 24c328a4e1c2419e9c2c7af8552c37e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24c328a4e1c2419e9c2c7af8552c37e7" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 33847 } }
I20260925 07:30:13.850154 9480 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 07:30:13.850440 14754 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24c328a4e1c2419e9c2c7af8552c37e7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "24c328a4e1c2419e9c2c7af8552c37e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24c328a4e1c2419e9c2c7af8552c37e7" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 33847 } } }
I20260925 07:30:13.850517 14754 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24c328a4e1c2419e9c2c7af8552c37e7 [sys.catalog]: This master's current role is: LEADER
I20260925 07:30:13.850672 14754 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24c328a4e1c2419e9c2c7af8552c37e7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 24c328a4e1c2419e9c2c7af8552c37e7. Latest consensus state: current_term: 1 leader_uuid: "24c328a4e1c2419e9c2c7af8552c37e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24c328a4e1c2419e9c2c7af8552c37e7" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 33847 } } }
I20260925 07:30:13.850735 14754 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24c328a4e1c2419e9c2c7af8552c37e7 [sys.catalog]: This master's current role is: LEADER
I20260925 07:30:13.851540 14761 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 07:30:13.852147 14761 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 07:30:13.853705 14761 catalog_manager.cc:1347] Generated new cluster ID: 9e94757245774cba8c41ec3af53fedba
I20260925 07:30:13.853760 14761 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 07:30:13.862632 14761 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 07:30:13.863317 14761 catalog_manager.cc:1504] Loading token signing keys...
I20260925 07:30:13.866394 14761 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 24c328a4e1c2419e9c2c7af8552c37e7: Generated new TSK 0
I20260925 07:30:13.866572 14761 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260925 07:30:13.895035 14779 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260925 07:30:13.895118 14779 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 07:30:13.895421 14752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:13.873206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48714 (local address 127.9.66.62:33847)
0925 07:30:13.873298 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:13.873300 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:13.873460 (+ 160us) server_negotiation.cc:407] Connection header received
0925 07:30:13.873495 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:13.873496 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:13.873508 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:13.873588 (+ 80us) server_negotiation.cc:226] Negotiated authn=JWT
0925 07:30:13.873802 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:13.874569 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:13.875323 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:13.875528 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:13.875625 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:13.875798 (+ 173us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 07:30:13.895192 (+ 19394us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39117/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0925 07:30:13.895361 (+ 169us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48714: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39117/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":68}
W20260925 07:30:13.895498 14752 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48714: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39117/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260925 07:30:13.895548 14778 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:30:13.873149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:33847 (local address 127.0.0.1:48714)
0925 07:30:13.873357 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:13.873366 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 07:30:13.873425 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:30:13.873597 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:30:13.873599 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:30:13.873601 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0925 07:30:13.873688 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:13.873690 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:13.874663 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:13.874664 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:13.875221 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:13.875223 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:13.875623 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:13.875626 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:13.875684 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:13.875688 (+ 4us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 07:30:13.895319 (+ 19631us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39117/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0925 07:30:13.895450 (+ 131us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.9.66.62:33847: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39117/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":121,"threads_started":1}
I20260925 07:30:13.895874 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 14682
2026-09-25T07:30:13Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (233 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260925 07:30:13.933208 9480 mini_oidc.cc:152] Starting JWKS server
I20260925 07:30:13.933969 9480 webserver.cc:460] Webserver started at https://127.0.0.1:33735/ using document root <none> and password file <none>
I20260925 07:30:13.934010 9480 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 07:30:13.934119 9480 webserver.cc:460] Webserver started at http://127.0.0.1:38157/ using document root <none> and password file <none>
2026-09-25T07:30:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T07:30:13Z Disabled control of system clock
I20260925 07:30:13.941680 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790321392278483-9480-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790321392278483-9480-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790321392278483-9480-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.66.62:38603
--webserver_interface=127.9.66.62
--webserver_port=0
--builtin_ntp_servers=127.9.66.20:41021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.66.62:38603
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33735/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260925 07:30:14.018829 14785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:30:14.019018 14785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:30:14.019037 14785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:30:14.020020 14785 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:30:14.020057 14785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:30:14.020071 14785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:30:14.020082 14785 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:30:14.020094 14785 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 07:30:14.020107 14785 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 07:30:14.020117 14785 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33735/jwks.json
I20260925 07:30:14.021322 14785 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:41021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790321392278483-9480-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790321392278483-9480-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.66.62:38603
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.66.62:38603
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33735/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.66.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14785
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790321392278483-9480-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:30:14.021487 14785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:30:14.021775 14785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:30:14.026280 14790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:30:14.026315 14791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:30:14.026283 14793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:30:14.026645 14785 server_base.cc:1034] running on GCE node
I20260925 07:30:14.026885 14785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:30:14.027164 14785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:30:14.028311 14785 hybrid_clock.cc:648] HybridClock initialized: now 1790321414028319 us; error 67 us; skew 500 ppm
I20260925 07:30:14.029627 14785 webserver.cc:460] Webserver started at http://127.9.66.62:42373/ using document root <none> and password file <none>
I20260925 07:30:14.029870 14785 fs_manager.cc:359] Metadata directory not provided
I20260925 07:30:14.029917 14785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:30:14.030059 14785 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:30:14.031035 14785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790321392278483-9480-0/minicluster-data/master-0/data/instance:
uuid: "b9b4bea9277d4dc1af2f0d55f6ef665c"
format_stamp: "Formatted at 2026-09-25 07:30:14 on dist-test-slave-57fn"
server_key: "1e0a131c2d4ded43c6beb86d7a6ed764"
server_key_iv: "5fb097486cd8aa53b2a926f53dae11ba"
server_key_version: "encryptionkey@0"
I20260925 07:30:14.031386 14785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790321392278483-9480-0/minicluster-data/master-0/wal/instance:
uuid: "b9b4bea9277d4dc1af2f0d55f6ef665c"
format_stamp: "Formatted at 2026-09-25 07:30:14 on dist-test-slave-57fn"
server_key: "1e0a131c2d4ded43c6beb86d7a6ed764"
server_key_iv: "5fb097486cd8aa53b2a926f53dae11ba"
server_key_version: "encryptionkey@0"
I20260925 07:30:14.032595 14785 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 07:30:14.033254 14799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:14.033399 14785 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 07:30:14.033478 14785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790321392278483-9480-0/minicluster-data/master-0/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790321392278483-9480-0/minicluster-data/master-0/wal
uuid: "b9b4bea9277d4dc1af2f0d55f6ef665c"
format_stamp: "Formatted at 2026-09-25 07:30:14 on dist-test-slave-57fn"
server_key: "1e0a131c2d4ded43c6beb86d7a6ed764"
server_key_iv: "5fb097486cd8aa53b2a926f53dae11ba"
server_key_version: "encryptionkey@0"
I20260925 07:30:14.033576 14785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790321392278483-9480-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790321392278483-9480-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790321392278483-9480-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:30:14.042485 14785 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:30:14.042696 14785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:30:14.042789 14785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:30:14.046361 14785 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.62:38603
I20260925 07:30:14.046389 14851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.62:38603 every 8 connection(s)
I20260925 07:30:14.046736 14785 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
I20260925 07:30:14.047405 14852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:30:14.049839 14852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9b4bea9277d4dc1af2f0d55f6ef665c: Bootstrap starting.
I20260925 07:30:14.050366 14852 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b9b4bea9277d4dc1af2f0d55f6ef665c: Neither blocks nor log segments found. Creating new log.
I20260925 07:30:14.050642 14852 log.cc:824] T 00000000000000000000000000000000 P b9b4bea9277d4dc1af2f0d55f6ef665c: Log is configured to *not* fsync() on all Append() calls
I20260925 07:30:14.051280 14852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9b4bea9277d4dc1af2f0d55f6ef665c: No bootstrap required, opened a new log
I20260925 07:30:14.052802 14852 raft_consensus.cc:348] T 00000000000000000000000000000000 P b9b4bea9277d4dc1af2f0d55f6ef665c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9b4bea9277d4dc1af2f0d55f6ef665c" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 38603 } }
I20260925 07:30:14.052954 14852 raft_consensus.cc:374] T 00000000000000000000000000000000 P b9b4bea9277d4dc1af2f0d55f6ef665c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:30:14.053000 14852 raft_consensus.cc:729] T 00000000000000000000000000000000 P b9b4bea9277d4dc1af2f0d55f6ef665c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9b4bea9277d4dc1af2f0d55f6ef665c, State: Initialized, Role: FOLLOWER
I20260925 07:30:14.053143 14852 consensus_queue.cc:260] T 00000000000000000000000000000000 P b9b4bea9277d4dc1af2f0d55f6ef665c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9b4bea9277d4dc1af2f0d55f6ef665c" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 38603 } }
I20260925 07:30:14.053231 14852 raft_consensus.cc:388] T 00000000000000000000000000000000 P b9b4bea9277d4dc1af2f0d55f6ef665c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 07:30:14.053283 14852 raft_consensus.cc:482] T 00000000000000000000000000000000 P b9b4bea9277d4dc1af2f0d55f6ef665c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 07:30:14.053331 14852 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b9b4bea9277d4dc1af2f0d55f6ef665c [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:30:14.053881 14852 raft_consensus.cc:504] T 00000000000000000000000000000000 P b9b4bea9277d4dc1af2f0d55f6ef665c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9b4bea9277d4dc1af2f0d55f6ef665c" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 38603 } }
I20260925 07:30:14.053994 14852 leader_election.cc:302] T 00000000000000000000000000000000 P b9b4bea9277d4dc1af2f0d55f6ef665c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b9b4bea9277d4dc1af2f0d55f6ef665c; no voters:
I20260925 07:30:14.054159 14852 leader_election.cc:288] T 00000000000000000000000000000000 P b9b4bea9277d4dc1af2f0d55f6ef665c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 07:30:14.054179 14855 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b9b4bea9277d4dc1af2f0d55f6ef665c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:30:14.054332 14855 raft_consensus.cc:686] T 00000000000000000000000000000000 P b9b4bea9277d4dc1af2f0d55f6ef665c [term 1 LEADER]: Becoming Leader. State: Replica: b9b4bea9277d4dc1af2f0d55f6ef665c, State: Running, Role: LEADER
I20260925 07:30:14.054399 14852 sys_catalog.cc:564] T 00000000000000000000000000000000 P b9b4bea9277d4dc1af2f0d55f6ef665c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:30:14.054430 14855 consensus_queue.cc:237] T 00000000000000000000000000000000 P b9b4bea9277d4dc1af2f0d55f6ef665c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9b4bea9277d4dc1af2f0d55f6ef665c" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 38603 } }
I20260925 07:30:14.054766 14856 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9b4bea9277d4dc1af2f0d55f6ef665c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b9b4bea9277d4dc1af2f0d55f6ef665c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9b4bea9277d4dc1af2f0d55f6ef665c" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 38603 } } }
I20260925 07:30:14.054870 14856 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9b4bea9277d4dc1af2f0d55f6ef665c [sys.catalog]: This master's current role is: LEADER
I20260925 07:30:14.054888 14857 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9b4bea9277d4dc1af2f0d55f6ef665c [sys.catalog]: SysCatalogTable state changed. Reason: New leader b9b4bea9277d4dc1af2f0d55f6ef665c. Latest consensus state: current_term: 1 leader_uuid: "b9b4bea9277d4dc1af2f0d55f6ef665c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9b4bea9277d4dc1af2f0d55f6ef665c" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 38603 } } }
I20260925 07:30:14.054943 14857 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9b4bea9277d4dc1af2f0d55f6ef665c [sys.catalog]: This master's current role is: LEADER
I20260925 07:30:14.055130 14862 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 07:30:14.055622 14862 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 07:30:14.055845 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 14785
I20260925 07:30:14.055931 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790321392278483-9480-0/minicluster-data/master-0/wal/instance
I20260925 07:30:14.056095 9480 external_mini_cluster.cc:1428] Setting key 342039360767c769ec9492475044fd4e
I20260925 07:30:14.057274 14862 catalog_manager.cc:1347] Generated new cluster ID: f720b089996c49d1a49440aa158da2cd
I20260925 07:30:14.057324 14862 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 07:30:14.061877 14874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:14.056487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40438 (local address 127.9.66.62:38603)
0925 07:30:14.056668 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:14.056676 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:14.056704 (+ 28us) server_negotiation.cc:407] Connection header received
0925 07:30:14.056813 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:14.056819 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:14.056883 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:14.057018 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:30:14.057356 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:14.058716 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:14.059773 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:14.060392 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:14.060519 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:14.061128 (+ 609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:30:14.061131 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:30:14.061132 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 07:30:14.061331 (+ 199us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:30:14.061371 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 07:30:14.061384 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:30:14.061451 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:14.061564 (+ 113us) server_negotiation.cc:299] Negotiation successful
0925 07:30:14.061638 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":70,"threads_started":1}
I20260925 07:30:14.062752 9480 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 07:30:14.065405 14862 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 07:30:14.065884 14862 catalog_manager.cc:1504] Loading token signing keys...
I20260925 07:30:14.071382 14862 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b9b4bea9277d4dc1af2f0d55f6ef665c: Generated new TSK 0
I20260925 07:30:14.071544 14862 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 07:30:14.081671 14874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:14.078176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40446 (local address 127.9.66.62:38603)
0925 07:30:14.078218 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:14.078220 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:14.078297 (+ 77us) server_negotiation.cc:407] Connection header received
0925 07:30:14.078388 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:14.078388 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:14.078397 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:14.078451 (+ 54us) server_negotiation.cc:226] Negotiated authn=JWT
0925 07:30:14.078626 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:14.079077 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:14.080197 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:14.080365 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:14.080412 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:14.080594 (+ 182us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 07:30:14.081424 (+ 830us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33735/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0925 07:30:14.081580 (+ 156us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40446: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33735/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":24}
W20260925 07:30:14.081727 14874 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40446: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33735/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260925 07:30:14.081765 14881 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:30:14.078071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:38603 (local address 127.0.0.1:40446)
0925 07:30:14.078243 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:14.078254 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 07:30:14.078312 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:30:14.078452 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:30:14.078453 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:30:14.078454 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0925 07:30:14.078511 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:14.078513 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:14.079180 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:14.079181 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:14.080115 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:14.080118 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:14.080434 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:14.080441 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:14.080504 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:14.080509 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 07:30:14.081545 (+ 1036us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33735/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0925 07:30:14.081627 (+ 82us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.9.66.62:38603: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33735/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":115,"threads_started":1}
I20260925 07:30:14.082003 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 14785
E20260925 07:30:14.095691 14882 webserver.cc:533] Webserver: sslize failed:
E20260925 07:30:14.095750 14882 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-25T07:30:14Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (185 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-25T07:30:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T07:30:14Z Disabled control of system clock
I20260925 07:30:14.126510 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790321392278483-9480-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790321392278483-9480-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790321392278483-9480-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.66.62:40101
--webserver_interface=127.9.66.62
--webserver_port=0
--builtin_ntp_servers=127.9.66.20:43179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.66.62:40101
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790321392278483-9480-0/testchainca.cert with env {}
W20260925 07:30:14.204687 14886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:30:14.204882 14886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:30:14.204903 14886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:30:14.206086 14886 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:30:14.206126 14886 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:30:14.206139 14886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:30:14.206195 14886 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:30:14.206247 14886 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 07:30:14.206281 14886 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 07:30:14.206305 14886 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260925 07:30:14.207787 14886 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:43179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790321392278483-9480-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790321392278483-9480-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.66.62:40101
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.66.62:40101
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.66.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790321392278483-9480-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14886
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790321392278483-9480-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:30:14.208038 14886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:30:14.208258 14886 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:30:14.213063 14892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:30:14.213088 14891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:30:14.213064 14894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:30:14.213182 14886 server_base.cc:1034] running on GCE node
I20260925 07:30:14.213542 14886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:30:14.213809 14886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:30:14.214943 14886 hybrid_clock.cc:648] HybridClock initialized: now 1790321414214930 us; error 29 us; skew 500 ppm
I20260925 07:30:14.216152 14886 webserver.cc:460] Webserver started at http://127.9.66.62:44941/ using document root <none> and password file <none>
I20260925 07:30:14.216382 14886 fs_manager.cc:359] Metadata directory not provided
I20260925 07:30:14.216424 14886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:30:14.216564 14886 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:30:14.217733 14886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790321392278483-9480-0/minicluster-data/master-0/data/instance:
uuid: "2d5cd09be41e45158919bf1158b9669b"
format_stamp: "Formatted at 2026-09-25 07:30:14 on dist-test-slave-57fn"
server_key: "ffa5fa7cc58014234126a79480135d37"
server_key_iv: "0e6b6f4c6e9bdb6d66324bf8ed33f693"
server_key_version: "encryptionkey@0"
I20260925 07:30:14.218076 14886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790321392278483-9480-0/minicluster-data/master-0/wal/instance:
uuid: "2d5cd09be41e45158919bf1158b9669b"
format_stamp: "Formatted at 2026-09-25 07:30:14 on dist-test-slave-57fn"
server_key: "ffa5fa7cc58014234126a79480135d37"
server_key_iv: "0e6b6f4c6e9bdb6d66324bf8ed33f693"
server_key_version: "encryptionkey@0"
I20260925 07:30:14.219383 14886 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 07:30:14.220095 14900 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:14.220278 14886 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 07:30:14.220359 14886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790321392278483-9480-0/minicluster-data/master-0/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790321392278483-9480-0/minicluster-data/master-0/wal
uuid: "2d5cd09be41e45158919bf1158b9669b"
format_stamp: "Formatted at 2026-09-25 07:30:14 on dist-test-slave-57fn"
server_key: "ffa5fa7cc58014234126a79480135d37"
server_key_iv: "0e6b6f4c6e9bdb6d66324bf8ed33f693"
server_key_version: "encryptionkey@0"
I20260925 07:30:14.220422 14886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790321392278483-9480-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790321392278483-9480-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790321392278483-9480-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:30:14.232815 14886 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:30:14.233085 14886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:30:14.233224 14886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:30:14.236809 14886 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.62:40101
I20260925 07:30:14.236865 14952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.62:40101 every 8 connection(s)
I20260925 07:30:14.237150 14886 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
I20260925 07:30:14.238125 14953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:30:14.240280 14953 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d5cd09be41e45158919bf1158b9669b: Bootstrap starting.
I20260925 07:30:14.240406 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 14886
I20260925 07:30:14.240530 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790321392278483-9480-0/minicluster-data/master-0/wal/instance
I20260925 07:30:14.240729 9480 external_mini_cluster.cc:1428] Setting key d58fd056efaa3e096b0c8dbeaa39771d
I20260925 07:30:14.241007 14953 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2d5cd09be41e45158919bf1158b9669b: Neither blocks nor log segments found. Creating new log.
I20260925 07:30:14.241374 14953 log.cc:824] T 00000000000000000000000000000000 P 2d5cd09be41e45158919bf1158b9669b: Log is configured to *not* fsync() on all Append() calls
I20260925 07:30:14.242157 14953 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d5cd09be41e45158919bf1158b9669b: No bootstrap required, opened a new log
I20260925 07:30:14.244459 14953 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2d5cd09be41e45158919bf1158b9669b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d5cd09be41e45158919bf1158b9669b" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 40101 } }
I20260925 07:30:14.244585 14953 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2d5cd09be41e45158919bf1158b9669b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:30:14.244606 14953 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2d5cd09be41e45158919bf1158b9669b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d5cd09be41e45158919bf1158b9669b, State: Initialized, Role: FOLLOWER
I20260925 07:30:14.244735 14953 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d5cd09be41e45158919bf1158b9669b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d5cd09be41e45158919bf1158b9669b" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 40101 } }
I20260925 07:30:14.244815 14953 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2d5cd09be41e45158919bf1158b9669b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 07:30:14.244848 14953 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2d5cd09be41e45158919bf1158b9669b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 07:30:14.244892 14953 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2d5cd09be41e45158919bf1158b9669b [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:30:14.245553 14953 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2d5cd09be41e45158919bf1158b9669b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d5cd09be41e45158919bf1158b9669b" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 40101 } }
I20260925 07:30:14.245653 14953 leader_election.cc:302] T 00000000000000000000000000000000 P 2d5cd09be41e45158919bf1158b9669b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2d5cd09be41e45158919bf1158b9669b; no voters:
I20260925 07:30:14.245818 14953 leader_election.cc:288] T 00000000000000000000000000000000 P 2d5cd09be41e45158919bf1158b9669b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 07:30:14.245838 14958 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2d5cd09be41e45158919bf1158b9669b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:30:14.245937 14958 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2d5cd09be41e45158919bf1158b9669b [term 1 LEADER]: Becoming Leader. State: Replica: 2d5cd09be41e45158919bf1158b9669b, State: Running, Role: LEADER
I20260925 07:30:14.246002 14958 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2d5cd09be41e45158919bf1158b9669b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d5cd09be41e45158919bf1158b9669b" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 40101 } }
I20260925 07:30:14.246052 14953 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2d5cd09be41e45158919bf1158b9669b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:30:14.246326 14958 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d5cd09be41e45158919bf1158b9669b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d5cd09be41e45158919bf1158b9669b. Latest consensus state: current_term: 1 leader_uuid: "2d5cd09be41e45158919bf1158b9669b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d5cd09be41e45158919bf1158b9669b" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 40101 } } }
I20260925 07:30:14.246395 14958 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d5cd09be41e45158919bf1158b9669b [sys.catalog]: This master's current role is: LEADER
I20260925 07:30:14.246592 14962 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 07:30:14.247241 14956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:14.241229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60198 (local address 127.9.66.62:40101)
0925 07:30:14.241440 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:14.241448 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:14.241477 (+ 29us) server_negotiation.cc:407] Connection header received
0925 07:30:14.242069 (+ 592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:14.242076 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:14.242146 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:14.242295 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:30:14.242488 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:14.243878 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:14.244811 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:14.245405 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:14.245546 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:14.246291 (+ 745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:30:14.246295 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:30:14.246295 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 07:30:14.246500 (+ 205us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:30:14.246539 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 07:30:14.246554 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:30:14.246615 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:14.246893 (+ 278us) server_negotiation.cc:299] Negotiation successful
0925 07:30:14.246978 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":77,"threads_started":1}
I20260925 07:30:14.247267 14959 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d5cd09be41e45158919bf1158b9669b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2d5cd09be41e45158919bf1158b9669b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d5cd09be41e45158919bf1158b9669b" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 40101 } } }
I20260925 07:30:14.247341 14959 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d5cd09be41e45158919bf1158b9669b [sys.catalog]: This master's current role is: LEADER
I20260925 07:30:14.247076 14962 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 07:30:14.248111 9480 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 07:30:14.250012 14962 catalog_manager.cc:1347] Generated new cluster ID: 72c82e47d4e14ccaad13a7c0e2c16f1c
I20260925 07:30:14.250069 14962 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 07:30:14.272295 14962 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 07:30:14.272969 14962 catalog_manager.cc:1504] Loading token signing keys...
I20260925 07:30:14.277966 14962 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2d5cd09be41e45158919bf1158b9669b: Generated new TSK 0
I20260925 07:30:14.278118 14962 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 07:30:14.370630 14956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:14.283662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60200 (local address 127.9.66.62:40101)
0925 07:30:14.283702 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:14.283705 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:14.283792 (+ 87us) server_negotiation.cc:407] Connection header received
0925 07:30:14.283863 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:14.283864 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:14.283872 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:14.283947 (+ 75us) server_negotiation.cc:226] Negotiated authn=JWT
0925 07:30:14.284090 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:14.284568 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:14.285704 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:14.285889 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:14.285939 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:14.286118 (+ 179us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 07:30:14.370422 (+ 84304us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0925 07:30:14.370572 (+ 150us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60200: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":22}
W20260925 07:30:14.370718 14956 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60200: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260925 07:30:14.370891 14982 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:30:14.283554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:40101 (local address 127.0.0.1:60200)
0925 07:30:14.283720 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:14.283731 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 07:30:14.283796 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:30:14.283970 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:30:14.283971 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:30:14.283974 (+ 3us) client_negotiation.cc:189] Negotiated authn=JWT
0925 07:30:14.284030 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:14.284033 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:14.284649 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:14.284650 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:14.285631 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:14.285634 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:14.285962 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:14.285969 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:14.286035 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:14.286040 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 07:30:14.370580 (+ 84540us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0925 07:30:14.370757 (+ 177us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.9.66.62:40101: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":97,"threads_started":1}
I20260925 07:30:14.371340 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 14886
2026-09-25T07:30:14Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (269 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (86 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (88 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790321392278483-9480-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 07:30:14 dist-test-slave-57fn krb5kdc[14994](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 07:30:14 dist-test-slave-57fn krb5kdc[14994](info): set up 2 sockets
Sep 25 07:30:14 dist-test-slave-57fn krb5kdc[14994](info): commencing operation
krb5kdc: starting...
W20260925 07:30:15.573880 9480 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790321392278483-9480-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790321392278483-9480-0/krb5kdc/test-user.keytab.
Sep 25 07:30:15 dist-test-slave-57fn krb5kdc[14994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321415, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T07:30:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T07:30:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.66.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.66.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.66.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab.
Entry for principal kudu/127.9.66.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab.
I20260925 07:30:15.692894 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790321392278483-9480-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790321392278483-9480-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790321392278483-9480-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.66.62:40973
--webserver_interface=127.9.66.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab
--principal=kudu/127.9.66.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:45057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.66.62:40973
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:30:15.772627 15007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:30:15.772886 15007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:30:15.772921 15007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:30:15.774091 15007 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:30:15.774163 15007 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:30:15.774186 15007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:30:15.774221 15007 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:30:15.774250 15007 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:30:15.775684 15007 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:45057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790321392278483-9480-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790321392278483-9480-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.66.62:40973
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab
--principal=kudu/127.9.66.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.66.62:40973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.66.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15007
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790321392278483-9480-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:30:15.775951 15007 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:30:15.776173 15007 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:30:15.780979 15015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:30:15.781009 15013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:30:15.781090 15007 server_base.cc:1034] running on GCE node
W20260925 07:30:15.780975 15012 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:30:15.781409 15007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:30:15.781781 15007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:30:15.782940 15007 hybrid_clock.cc:648] HybridClock initialized: now 1790321415782927 us; error 29 us; skew 500 ppm
Sep 25 07:30:15 dist-test-slave-57fn krb5kdc[14994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321415, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:30:15.792279 15007 init.cc:377] Logged in from keytab as kudu/127.9.66.62@KRBTEST.COM (short username kudu)
I20260925 07:30:15.792675 15007 webserver.cc:460] Webserver started at http://127.9.66.62:37055/ using document root <none> and password file <none>
I20260925 07:30:15.792877 15007 fs_manager.cc:359] Metadata directory not provided
I20260925 07:30:15.792918 15007 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:30:15.793010 15007 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:30:15.794081 15007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790321392278483-9480-0/minicluster-data/master-0/data/instance:
uuid: "544d8117c0de4fe3919cc46537ab005b"
format_stamp: "Formatted at 2026-09-25 07:30:15 on dist-test-slave-57fn"
server_key: "7db0977d9bec36ce795da4d7ed3ddff0"
server_key_iv: "c300c6d95e32bc9999224e46db9ce0af"
server_key_version: "encryptionkey@0"
I20260925 07:30:15.794389 15007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790321392278483-9480-0/minicluster-data/master-0/wal/instance:
uuid: "544d8117c0de4fe3919cc46537ab005b"
format_stamp: "Formatted at 2026-09-25 07:30:15 on dist-test-slave-57fn"
server_key: "7db0977d9bec36ce795da4d7ed3ddff0"
server_key_iv: "c300c6d95e32bc9999224e46db9ce0af"
server_key_version: "encryptionkey@0"
I20260925 07:30:15.795648 15007 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 07:30:15.796306 15022 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:15.796479 15007 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 07:30:15.796574 15007 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790321392278483-9480-0/minicluster-data/master-0/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790321392278483-9480-0/minicluster-data/master-0/wal
uuid: "544d8117c0de4fe3919cc46537ab005b"
format_stamp: "Formatted at 2026-09-25 07:30:15 on dist-test-slave-57fn"
server_key: "7db0977d9bec36ce795da4d7ed3ddff0"
server_key_iv: "c300c6d95e32bc9999224e46db9ce0af"
server_key_version: "encryptionkey@0"
I20260925 07:30:15.796648 15007 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790321392278483-9480-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790321392278483-9480-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790321392278483-9480-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:30:15.818482 15007 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:30:15.820943 15007 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:30:15.821204 15007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:30:15.825093 15007 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.62:40973
I20260925 07:30:15.825141 15074 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.62:40973 every 8 connection(s)
I20260925 07:30:15.825565 15007 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
I20260925 07:30:15.826370 15075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:30:15.827343 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 15007
I20260925 07:30:15.827472 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790321392278483-9480-0/minicluster-data/master-0/wal/instance
I20260925 07:30:15.827674 9480 external_mini_cluster.cc:1428] Setting key 579abd57b1c61ce453778efdc717f5da
I20260925 07:30:15.829692 15075 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 544d8117c0de4fe3919cc46537ab005b: Bootstrap starting.
I20260925 07:30:15.830276 15075 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 544d8117c0de4fe3919cc46537ab005b: Neither blocks nor log segments found. Creating new log.
I20260925 07:30:15.830539 15075 log.cc:824] T 00000000000000000000000000000000 P 544d8117c0de4fe3919cc46537ab005b: Log is configured to *not* fsync() on all Append() calls
I20260925 07:30:15.831198 15075 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 544d8117c0de4fe3919cc46537ab005b: No bootstrap required, opened a new log
I20260925 07:30:15.832903 15075 raft_consensus.cc:348] T 00000000000000000000000000000000 P 544d8117c0de4fe3919cc46537ab005b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "544d8117c0de4fe3919cc46537ab005b" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 40973 } }
I20260925 07:30:15.833024 15075 raft_consensus.cc:374] T 00000000000000000000000000000000 P 544d8117c0de4fe3919cc46537ab005b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:30:15.833046 15075 raft_consensus.cc:729] T 00000000000000000000000000000000 P 544d8117c0de4fe3919cc46537ab005b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 544d8117c0de4fe3919cc46537ab005b, State: Initialized, Role: FOLLOWER
I20260925 07:30:15.833171 15075 consensus_queue.cc:260] T 00000000000000000000000000000000 P 544d8117c0de4fe3919cc46537ab005b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "544d8117c0de4fe3919cc46537ab005b" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 40973 } }
I20260925 07:30:15.833240 15075 raft_consensus.cc:388] T 00000000000000000000000000000000 P 544d8117c0de4fe3919cc46537ab005b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 07:30:15.833261 15075 raft_consensus.cc:482] T 00000000000000000000000000000000 P 544d8117c0de4fe3919cc46537ab005b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 07:30:15.833328 15075 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 544d8117c0de4fe3919cc46537ab005b [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:30:15.833904 15075 raft_consensus.cc:504] T 00000000000000000000000000000000 P 544d8117c0de4fe3919cc46537ab005b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "544d8117c0de4fe3919cc46537ab005b" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 40973 } }
Sep 25 07:30:15 dist-test-slave-57fn krb5kdc[14994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321415, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.66.62@KRBTEST.COM
I20260925 07:30:15.834019 15075 leader_election.cc:302] T 00000000000000000000000000000000 P 544d8117c0de4fe3919cc46537ab005b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 544d8117c0de4fe3919cc46537ab005b; no voters:
I20260925 07:30:15.834172 15075 leader_election.cc:288] T 00000000000000000000000000000000 P 544d8117c0de4fe3919cc46537ab005b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 07:30:15.834223 15080 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 544d8117c0de4fe3919cc46537ab005b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:30:15.834352 15080 raft_consensus.cc:686] T 00000000000000000000000000000000 P 544d8117c0de4fe3919cc46537ab005b [term 1 LEADER]: Becoming Leader. State: Replica: 544d8117c0de4fe3919cc46537ab005b, State: Running, Role: LEADER
I20260925 07:30:15.834420 15075 sys_catalog.cc:564] T 00000000000000000000000000000000 P 544d8117c0de4fe3919cc46537ab005b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:30:15.834436 15080 consensus_queue.cc:237] T 00000000000000000000000000000000 P 544d8117c0de4fe3919cc46537ab005b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "544d8117c0de4fe3919cc46537ab005b" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 40973 } }
I20260925 07:30:15.835004 15080 sys_catalog.cc:455] T 00000000000000000000000000000000 P 544d8117c0de4fe3919cc46537ab005b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 544d8117c0de4fe3919cc46537ab005b. Latest consensus state: current_term: 1 leader_uuid: "544d8117c0de4fe3919cc46537ab005b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "544d8117c0de4fe3919cc46537ab005b" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 40973 } } }
I20260925 07:30:15.835086 15080 sys_catalog.cc:458] T 00000000000000000000000000000000 P 544d8117c0de4fe3919cc46537ab005b [sys.catalog]: This master's current role is: LEADER
I20260925 07:30:15.835309 15089 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 07:30:15.835579 15082 sys_catalog.cc:455] T 00000000000000000000000000000000 P 544d8117c0de4fe3919cc46537ab005b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "544d8117c0de4fe3919cc46537ab005b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "544d8117c0de4fe3919cc46537ab005b" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 40973 } } }
I20260925 07:30:15.835654 15082 sys_catalog.cc:458] T 00000000000000000000000000000000 P 544d8117c0de4fe3919cc46537ab005b [sys.catalog]: This master's current role is: LEADER
I20260925 07:30:15.835786 15089 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 07:30:15.837888 15089 catalog_manager.cc:1347] Generated new cluster ID: 845d2e7974424987a80caf034fe1f07d
I20260925 07:30:15.838045 15089 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 07:30:15.838192 15078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:15.828176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54958 (local address 127.9.66.62:40973)
0925 07:30:15.828337 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:15.828350 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:15.828373 (+ 23us) server_negotiation.cc:407] Connection header received
0925 07:30:15.828755 (+ 382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:15.828761 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:15.828805 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:15.828913 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:30:15.829729 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:15.830742 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:15.831544 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:15.831946 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:15.832067 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:15.834472 (+ 2405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:30:15.834475 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:30:15.834476 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:30:15.834488 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:30:15.836516 (+ 2028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:15.836901 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:15.836903 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:15.836903 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:15.836998 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:15.837237 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:15.837238 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:15.837239 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:15.837544 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:30:15.837673 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:15.837803 (+ 130us) server_negotiation.cc:299] Negotiation successful
0925 07:30:15.837937 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":85,"threads_started":1}
I20260925 07:30:15.847548 15089 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 07:30:15.848073 15089 catalog_manager.cc:1504] Loading token signing keys...
I20260925 07:30:15.853659 15089 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 544d8117c0de4fe3919cc46537ab005b: Generated new TSK 0
I20260925 07:30:15.853818 15089 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.66.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.66.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.66.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.1.keytab.
Entry for principal kudu/127.9.66.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.1.keytab.
I20260925 07:30:15.879912 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790321392278483-9480-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790321392278483-9480-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790321392278483-9480-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790321392278483-9480-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.66.1:0
--local_ip_for_outbound_sockets=127.9.66.1
--webserver_interface=127.9.66.1
--webserver_port=0
--tserver_master_addrs=127.9.66.62:40973
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.1.keytab
--principal=kudu/127.9.66.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:45057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:30:15.958149 15101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:30:15.958377 15101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:30:15.958410 15101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:30:15.959477 15101 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:30:15.959549 15101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:30:15.959589 15101 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.66.1
I20260925 07:30:15.960994 15101 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:45057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790321392278483-9480-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790321392278483-9480-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.1.keytab
--principal=kudu/127.9.66.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.66.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790321392278483-9480-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.66.1
--webserver_port=0
--tserver_master_addrs=127.9.66.62:40973
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15101
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.66.1
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790321392278483-9480-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:30:15.961265 15101 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:30:15.961477 15101 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:30:15.966289 15107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:30:15.966289 15109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:30:15.966305 15106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:30:15.966789 15101 server_base.cc:1034] running on GCE node
I20260925 07:30:15.967010 15101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:30:15.967209 15101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:30:15.968348 15101 hybrid_clock.cc:648] HybridClock initialized: now 1790321415968330 us; error 31 us; skew 500 ppm
Sep 25 07:30:15 dist-test-slave-57fn krb5kdc[14994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321415, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:30:15.970865 15101 init.cc:377] Logged in from keytab as kudu/127.9.66.1@KRBTEST.COM (short username kudu)
I20260925 07:30:15.971302 15101 webserver.cc:460] Webserver started at http://127.9.66.1:43825/ using document root <none> and password file <none>
I20260925 07:30:15.971544 15101 fs_manager.cc:359] Metadata directory not provided
I20260925 07:30:15.971591 15101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:30:15.971761 15101 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:30:15.972803 15101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790321392278483-9480-0/minicluster-data/ts-0/data/instance:
uuid: "a9e5ca53081449498feadd6197e833a6"
format_stamp: "Formatted at 2026-09-25 07:30:15 on dist-test-slave-57fn"
server_key: "489ed5da94db5ec14837b82c5cef2ca8"
server_key_iv: "403f5472f118bd40e0be628ca5b9b47f"
server_key_version: "encryptionkey@0"
I20260925 07:30:15.973207 15101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790321392278483-9480-0/minicluster-data/ts-0/wal/instance:
uuid: "a9e5ca53081449498feadd6197e833a6"
format_stamp: "Formatted at 2026-09-25 07:30:15 on dist-test-slave-57fn"
server_key: "489ed5da94db5ec14837b82c5cef2ca8"
server_key_iv: "403f5472f118bd40e0be628ca5b9b47f"
server_key_version: "encryptionkey@0"
I20260925 07:30:15.974704 15101 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 07:30:15.975454 15116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:15.975607 15101 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 07:30:15.975704 15101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790321392278483-9480-0/minicluster-data/ts-0/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790321392278483-9480-0/minicluster-data/ts-0/wal
uuid: "a9e5ca53081449498feadd6197e833a6"
format_stamp: "Formatted at 2026-09-25 07:30:15 on dist-test-slave-57fn"
server_key: "489ed5da94db5ec14837b82c5cef2ca8"
server_key_iv: "403f5472f118bd40e0be628ca5b9b47f"
server_key_version: "encryptionkey@0"
I20260925 07:30:15.975781 15101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790321392278483-9480-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790321392278483-9480-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790321392278483-9480-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:30:15.996002 15101 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:30:15.997928 15101 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:30:15.998086 15101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:30:15.998320 15101 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:30:15.998652 15101 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:30:15.998710 15101 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:15.998750 15101 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:30:15.998778 15101 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:16.004676 15101 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.1:34967
I20260925 07:30:16.004730 15229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.1:34967 every 8 connection(s)
I20260925 07:30:16.005026 15101 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790321392278483-9480-0/minicluster-data/ts-0/data/info.pb
Sep 25 07:30:16 dist-test-slave-57fn krb5kdc[14994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321415, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.1@KRBTEST.COM for kudu/127.9.66.62@KRBTEST.COM
I20260925 07:30:16.014060 15078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:16.005794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.1:34703 (local address 127.9.66.62:40973)
0925 07:30:16.005841 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:16.005843 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:16.006023 (+ 180us) server_negotiation.cc:407] Connection header received
0925 07:30:16.006902 (+ 879us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:16.006903 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:16.006912 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:16.006968 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:30:16.007601 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:16.008132 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:16.009092 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:16.009301 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:16.009359 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:16.011501 (+ 2142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:30:16.011502 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:30:16.011502 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:30:16.011508 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:30:16.012924 (+ 1416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:16.013267 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:16.013268 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:16.013268 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:16.013316 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:16.013554 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:16.013555 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:16.013555 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:16.013662 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:30:16.013714 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:16.013920 (+ 206us) server_negotiation.cc:299] Negotiation successful
0925 07:30:16.013985 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 07:30:16.014360 15232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:16.005740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:40973 (local address 127.9.66.1:34703)
0925 07:30:16.005899 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:16.005938 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 07:30:16.006818 (+ 880us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:30:16.007033 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:30:16.007036 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:30:16.007280 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:30:16.007488 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:16.007491 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:16.008229 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:16.008230 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:16.009013 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:16.009015 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:16.009377 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:16.009378 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:16.009471 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:16.010011 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:30:16.010021 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:30:16.011418 (+ 1397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:30:16.012993 (+ 1575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:30:16.012997 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:30:16.012999 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:30:16.013197 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:30:16.013372 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:30:16.013372 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:30:16.013373 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:30:16.013459 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:30:16.013721 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:30:16.013725 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:30:16.013827 (+ 102us) client_negotiation.cc:769] Sending connection context
0925 07:30:16.013908 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 07:30:16.014030 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":96,"threads_started":1}
I20260925 07:30:16.014695 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 15101
I20260925 07:30:16.014827 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790321392278483-9480-0/minicluster-data/ts-0/wal/instance
I20260925 07:30:16.014969 15230 heartbeater.cc:347] Connected to a master server at 127.9.66.62:40973
I20260925 07:30:16.015014 9480 external_mini_cluster.cc:1428] Setting key 62b4fff0bef174eb621d920676c50682
I20260925 07:30:16.015092 15230 heartbeater.cc:464] Registering TS with master...
I20260925 07:30:16.015309 15230 heartbeater.cc:511] Master 127.9.66.62:40973 requested a full tablet report, sending...
I20260925 07:30:16.015931 15039 ts_manager.cc:194] Registered new tserver with Master: a9e5ca53081449498feadd6197e833a6 (127.9.66.1:34967)
I20260925 07:30:16.016820 15039 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.66.1@KRBTEST.COM'} at 127.9.66.1:34703
WARNING: no policy specified for kudu/127.9.66.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.66.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.66.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.2.keytab.
Entry for principal kudu/127.9.66.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.2.keytab.
I20260925 07:30:16.040337 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790321392278483-9480-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790321392278483-9480-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790321392278483-9480-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790321392278483-9480-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.66.2:0
--local_ip_for_outbound_sockets=127.9.66.2
--webserver_interface=127.9.66.2
--webserver_port=0
--tserver_master_addrs=127.9.66.62:40973
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.2.keytab
--principal=kudu/127.9.66.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:45057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:30:16.123032 15235 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:30:16.123260 15235 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:30:16.123294 15235 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:30:16.124403 15235 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:30:16.124455 15235 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:30:16.124495 15235 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.66.2
I20260925 07:30:16.125864 15235 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:45057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790321392278483-9480-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790321392278483-9480-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.2.keytab
--principal=kudu/127.9.66.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.66.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790321392278483-9480-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.66.2
--webserver_port=0
--tserver_master_addrs=127.9.66.62:40973
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15235
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.66.2
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790321392278483-9480-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:30:16.126127 15235 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:30:16.126331 15235 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:30:16.131414 15241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:30:16.131476 15235 server_base.cc:1034] running on GCE node
W20260925 07:30:16.131413 15240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:30:16.131430 15243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:30:16.131831 15235 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:30:16.132090 15235 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:30:16.133256 15235 hybrid_clock.cc:648] HybridClock initialized: now 1790321416133235 us; error 38 us; skew 500 ppm
Sep 25 07:30:16 dist-test-slave-57fn krb5kdc[14994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321416, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:30:16.135955 15235 init.cc:377] Logged in from keytab as kudu/127.9.66.2@KRBTEST.COM (short username kudu)
I20260925 07:30:16.136418 15235 webserver.cc:460] Webserver started at http://127.9.66.2:42701/ using document root <none> and password file <none>
I20260925 07:30:16.136642 15235 fs_manager.cc:359] Metadata directory not provided
I20260925 07:30:16.136710 15235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:30:16.136852 15235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:30:16.137917 15235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790321392278483-9480-0/minicluster-data/ts-1/data/instance:
uuid: "cf79ed298de64dc59a15b0ec056b599b"
format_stamp: "Formatted at 2026-09-25 07:30:16 on dist-test-slave-57fn"
server_key: "61c87c77636680589336e02a43584db0"
server_key_iv: "50234775b774db06b4c761e148b84ff2"
server_key_version: "encryptionkey@0"
I20260925 07:30:16.138258 15235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790321392278483-9480-0/minicluster-data/ts-1/wal/instance:
uuid: "cf79ed298de64dc59a15b0ec056b599b"
format_stamp: "Formatted at 2026-09-25 07:30:16 on dist-test-slave-57fn"
server_key: "61c87c77636680589336e02a43584db0"
server_key_iv: "50234775b774db06b4c761e148b84ff2"
server_key_version: "encryptionkey@0"
I20260925 07:30:16.139514 15235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 07:30:16.140209 15250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:16.140388 15235 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 07:30:16.140484 15235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790321392278483-9480-0/minicluster-data/ts-1/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790321392278483-9480-0/minicluster-data/ts-1/wal
uuid: "cf79ed298de64dc59a15b0ec056b599b"
format_stamp: "Formatted at 2026-09-25 07:30:16 on dist-test-slave-57fn"
server_key: "61c87c77636680589336e02a43584db0"
server_key_iv: "50234775b774db06b4c761e148b84ff2"
server_key_version: "encryptionkey@0"
I20260925 07:30:16.140563 15235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790321392278483-9480-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790321392278483-9480-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790321392278483-9480-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:30:16.165275 15235 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:30:16.166951 15235 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:30:16.167097 15235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:30:16.167322 15235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:30:16.167618 15235 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:30:16.167675 15235 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:16.167714 15235 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:30:16.167739 15235 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:16.173650 15235 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.2:44981
I20260925 07:30:16.173694 15363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.2:44981 every 8 connection(s)
I20260925 07:30:16.174037 15235 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790321392278483-9480-0/minicluster-data/ts-1/data/info.pb
I20260925 07:30:16.174952 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 15235
I20260925 07:30:16.175033 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790321392278483-9480-0/minicluster-data/ts-1/wal/instance
I20260925 07:30:16.175251 9480 external_mini_cluster.cc:1428] Setting key 4be2565d494caa72b91cca006972679a
Sep 25 07:30:16 dist-test-slave-57fn krb5kdc[14994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321416, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.2@KRBTEST.COM for kudu/127.9.66.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.66.3@KRBTEST.COM; defaulting to no policy
I20260925 07:30:16.183540 15078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:16.174751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.2:54487 (local address 127.9.66.62:40973)
0925 07:30:16.174820 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:16.174822 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:16.175030 (+ 208us) server_negotiation.cc:407] Connection header received
0925 07:30:16.176212 (+ 1182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:16.176213 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:16.176224 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:16.176260 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:30:16.176851 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:16.177335 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:16.178400 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:16.178657 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:16.178737 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:16.180637 (+ 1900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:30:16.180638 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:30:16.180639 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:30:16.180664 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:30:16.182149 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:16.182597 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:16.182599 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:16.182600 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:16.182658 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:16.182925 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:16.182926 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:16.182926 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:16.183035 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:30:16.183113 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:16.183408 (+ 295us) server_negotiation.cc:299] Negotiation successful
0925 07:30:16.183478 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 07:30:16.183750 15366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:16.174721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:40973 (local address 127.9.66.2:54487)
0925 07:30:16.174895 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:16.174934 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 07:30:16.176116 (+ 1182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:30:16.176346 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:30:16.176350 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:30:16.176587 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:30:16.176756 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:16.176760 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:16.177448 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:16.177449 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:16.178311 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:16.178313 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:16.178738 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:16.178739 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:16.178861 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:16.179328 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:30:16.179337 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:30:16.180547 (+ 1210us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:30:16.182260 (+ 1713us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:30:16.182267 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:30:16.182269 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:30:16.182505 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:30:16.182744 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:30:16.182745 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:30:16.182745 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:30:16.182848 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:30:16.183103 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:30:16.183109 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:30:16.183252 (+ 143us) client_negotiation.cc:769] Sending connection context
0925 07:30:16.183376 (+ 124us) client_negotiation.cc:240] Negotiation successful
0925 07:30:16.183510 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":105,"threads_started":1}
I20260925 07:30:16.184288 15364 heartbeater.cc:347] Connected to a master server at 127.9.66.62:40973
I20260925 07:30:16.184408 15364 heartbeater.cc:464] Registering TS with master...
I20260925 07:30:16.184567 15364 heartbeater.cc:511] Master 127.9.66.62:40973 requested a full tablet report, sending...
I20260925 07:30:16.184952 15039 ts_manager.cc:194] Registered new tserver with Master: cf79ed298de64dc59a15b0ec056b599b (127.9.66.2:44981)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.66.3@KRBTEST.COM" created.
I20260925 07:30:16.185412 15039 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.66.2@KRBTEST.COM'} at 127.9.66.2:54487
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.66.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.3.keytab.
Entry for principal kudu/127.9.66.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.3.keytab.
I20260925 07:30:16.200161 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790321392278483-9480-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790321392278483-9480-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790321392278483-9480-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790321392278483-9480-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.66.3:0
--local_ip_for_outbound_sockets=127.9.66.3
--webserver_interface=127.9.66.3
--webserver_port=0
--tserver_master_addrs=127.9.66.62:40973
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.3.keytab
--principal=kudu/127.9.66.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:45057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:30:16.279814 15369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:30:16.280050 15369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:30:16.280088 15369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:30:16.281364 15369 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:30:16.281450 15369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:30:16.281498 15369 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.66.3
I20260925 07:30:16.283354 15369 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:45057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790321392278483-9480-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790321392278483-9480-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.3.keytab
--principal=kudu/127.9.66.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.66.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790321392278483-9480-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.66.3
--webserver_port=0
--tserver_master_addrs=127.9.66.62:40973
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15369
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.66.3
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790321392278483-9480-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:30:16.283627 15369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:30:16.283875 15369 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:30:16.289181 15377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:30:16.289201 15375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:30:16.289181 15374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:30:16.289471 15369 server_base.cc:1034] running on GCE node
I20260925 07:30:16.289716 15369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:30:16.289963 15369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:30:16.291147 15369 hybrid_clock.cc:648] HybridClock initialized: now 1790321416291130 us; error 27 us; skew 500 ppm
Sep 25 07:30:16 dist-test-slave-57fn krb5kdc[14994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321416, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:30:16.293776 15369 init.cc:377] Logged in from keytab as kudu/127.9.66.3@KRBTEST.COM (short username kudu)
I20260925 07:30:16.294251 15369 webserver.cc:460] Webserver started at http://127.9.66.3:41637/ using document root <none> and password file <none>
I20260925 07:30:16.294482 15369 fs_manager.cc:359] Metadata directory not provided
I20260925 07:30:16.294569 15369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:30:16.294703 15369 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:30:16.295732 15369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790321392278483-9480-0/minicluster-data/ts-2/data/instance:
uuid: "0493e8c61dd2473ba694197f6456ac4d"
format_stamp: "Formatted at 2026-09-25 07:30:16 on dist-test-slave-57fn"
server_key: "c934c9ba3c443a2ada55fcbea26c3e15"
server_key_iv: "1fc49ad5a04ddc31373d0e23cf168eac"
server_key_version: "encryptionkey@0"
I20260925 07:30:16.296190 15369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790321392278483-9480-0/minicluster-data/ts-2/wal/instance:
uuid: "0493e8c61dd2473ba694197f6456ac4d"
format_stamp: "Formatted at 2026-09-25 07:30:16 on dist-test-slave-57fn"
server_key: "c934c9ba3c443a2ada55fcbea26c3e15"
server_key_iv: "1fc49ad5a04ddc31373d0e23cf168eac"
server_key_version: "encryptionkey@0"
I20260925 07:30:16.297708 15369 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 07:30:16.298484 15384 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:16.298664 15369 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 07:30:16.298764 15369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790321392278483-9480-0/minicluster-data/ts-2/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790321392278483-9480-0/minicluster-data/ts-2/wal
uuid: "0493e8c61dd2473ba694197f6456ac4d"
format_stamp: "Formatted at 2026-09-25 07:30:16 on dist-test-slave-57fn"
server_key: "c934c9ba3c443a2ada55fcbea26c3e15"
server_key_iv: "1fc49ad5a04ddc31373d0e23cf168eac"
server_key_version: "encryptionkey@0"
I20260925 07:30:16.298852 15369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790321392278483-9480-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790321392278483-9480-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790321392278483-9480-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:30:16.318637 15369 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:30:16.320931 15369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:30:16.321122 15369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:30:16.321383 15369 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:30:16.321751 15369 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:30:16.321790 15369 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:16.321841 15369 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:30:16.321873 15369 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:16.328243 15369 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.3:44505
I20260925 07:30:16.328275 15497 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.3:44505 every 8 connection(s)
I20260925 07:30:16.328629 15369 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790321392278483-9480-0/minicluster-data/ts-2/data/info.pb
I20260925 07:30:16.334877 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 15369
I20260925 07:30:16.334966 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790321392278483-9480-0/minicluster-data/ts-2/wal/instance
Sep 25 07:30:16 dist-test-slave-57fn krb5kdc[14994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321416, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.3@KRBTEST.COM for kudu/127.9.66.62@KRBTEST.COM
I20260925 07:30:16.335247 9480 external_mini_cluster.cc:1428] Setting key e31ee390166e1000f07fd6948846143f
I20260925 07:30:16.338614 15078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:16.329436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.3:56189 (local address 127.9.66.62:40973)
0925 07:30:16.329528 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:16.329530 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:16.329764 (+ 234us) server_negotiation.cc:407] Connection header received
0925 07:30:16.330689 (+ 925us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:16.330691 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:16.330700 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:16.330756 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:30:16.331372 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:16.331892 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:16.332883 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:16.333068 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:16.333125 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:16.335714 (+ 2589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:30:16.335715 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:30:16.335716 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:30:16.335723 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:30:16.337168 (+ 1445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:16.337607 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:16.337608 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:16.337608 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:16.337668 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:16.337957 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:16.337958 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:16.337958 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:16.338103 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:30:16.338176 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:16.338457 (+ 281us) server_negotiation.cc:299] Negotiation successful
0925 07:30:16.338529 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 07:30:16.338795 15500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:16.329431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:40973 (local address 127.9.66.3:56189)
0925 07:30:16.329639 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:16.329676 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 07:30:16.330590 (+ 914us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:30:16.330809 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:30:16.330812 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:30:16.331088 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:30:16.331288 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:16.331293 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:16.331995 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:16.331996 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:16.332799 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:16.332801 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:16.333151 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:16.333152 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:16.333253 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:16.333816 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:30:16.333824 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:30:16.335596 (+ 1772us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:30:16.337284 (+ 1688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:30:16.337290 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:30:16.337292 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:30:16.337547 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:30:16.337778 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:30:16.337778 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:30:16.337779 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:30:16.337886 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:30:16.338179 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:30:16.338185 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:30:16.338316 (+ 131us) client_negotiation.cc:769] Sending connection context
0925 07:30:16.338421 (+ 105us) client_negotiation.cc:240] Negotiation successful
0925 07:30:16.338563 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":111,"threads_started":1}
I20260925 07:30:16.339152 15498 heartbeater.cc:347] Connected to a master server at 127.9.66.62:40973
I20260925 07:30:16.339257 15498 heartbeater.cc:464] Registering TS with master...
I20260925 07:30:16.339445 15498 heartbeater.cc:511] Master 127.9.66.62:40973 requested a full tablet report, sending...
I20260925 07:30:16.339820 15039 ts_manager.cc:194] Registered new tserver with Master: 0493e8c61dd2473ba694197f6456ac4d (127.9.66.3:44505)
I20260925 07:30:16.340201 15039 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.66.3@KRBTEST.COM'} at 127.9.66.3:56189
I20260925 07:30:16.346987 9480 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790321392278483-9480-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790321392278483-9480-0/krb5kdc/test-admin.keytab.
Sep 25 07:30:16 dist-test-slave-57fn krb5kdc[14994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321416, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:30:16.361990 9480 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260925 07:30:16.363588 15505 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:30:16.362831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36478 (local address 127.9.66.1:34967)
0925 07:30:16.363009 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:16.363013 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:16.363023 (+ 10us) server_negotiation.cc:407] Connection header received
0925 07:30:16.363047 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:16.363049 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:16.363062 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:16.363143 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:30:16.363501 (+ 358us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36478: BlockingRecv error: recv got EOF from 127.0.0.1:36478 (error 108)
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":95,"threads_started":1}
W20260925 07:30:16.363668 9480 init.cc:244] Error while opening credential cache '/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790321392278483-9480-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790321392278483-9480-0/krb5kdc/krb5cc)
Sep 25 07:30:16 dist-test-slave-57fn krb5kdc[14994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321416, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:30:16.364686 9480 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 07:30:16 dist-test-slave-57fn krb5kdc[14994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321416, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:30:16.365612 9480 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 07:30:16 dist-test-slave-57fn krb5kdc[14994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321416, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.66.1@KRBTEST.COM
I20260925 07:30:16.373006 15505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:16.366145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36490 (local address 127.9.66.1:34967)
0925 07:30:16.366208 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:16.366210 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:16.366220 (+ 10us) server_negotiation.cc:407] Connection header received
0925 07:30:16.366278 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:16.366278 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:16.366287 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:16.366332 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:30:16.366672 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:16.367450 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:16.368116 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:16.368416 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:16.368466 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:16.370527 (+ 2061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:30:16.370533 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:30:16.370534 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:30:16.370540 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:30:16.372112 (+ 1572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:16.372410 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:16.372412 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:16.372412 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:16.372463 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:16.372606 (+ 143us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:16.372606 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:16.372607 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:16.372712 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:30:16.372824 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:16.372883 (+ 59us) server_negotiation.cc:299] Negotiation successful
0925 07:30:16.372940 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
W20260925 07:30:16.374677 15505 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:30:16.374231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36498 (local address 127.9.66.1:34967)
0925 07:30:16.374281 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:16.374282 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:16.374296 (+ 14us) server_negotiation.cc:407] Connection header received
0925 07:30:16.374332 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:16.374332 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:16.374339 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:16.374400 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:30:16.374625 (+ 225us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36498: BlockingRecv error: recv got EOF from 127.0.0.1:36498 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20260925 07:30:16.374866 9480 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790321392278483-9480-0/krb5kdc/krb5cc' of type FILE
Sep 25 07:30:16 dist-test-slave-57fn krb5kdc[14994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321416, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:30:16.375785 9480 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 07:30:16 dist-test-slave-57fn krb5kdc[14994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321416, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:30:16.376573 9480 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 07:30:16 dist-test-slave-57fn krb5kdc[14994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321416, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.66.1@KRBTEST.COM
I20260925 07:30:16.383044 15505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:16.377045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36510 (local address 127.9.66.1:34967)
0925 07:30:16.377102 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:16.377104 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:16.377132 (+ 28us) server_negotiation.cc:407] Connection header received
0925 07:30:16.377200 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:16.377201 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:16.377209 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:16.377260 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:30:16.377686 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:16.378220 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:16.378906 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:16.379097 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:16.379201 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:16.381000 (+ 1799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:30:16.381001 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:30:16.381001 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:30:16.381006 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:30:16.382200 (+ 1194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:16.382485 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:16.382486 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:16.382486 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:16.382531 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:16.382684 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:16.382684 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:16.382685 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:16.382774 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:30:16.382840 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:16.382936 (+ 96us) server_negotiation.cc:299] Negotiation successful
0925 07:30:16.382989 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
W20260925 07:30:16.384400 15505 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:30:16.383986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36516 (local address 127.9.66.1:34967)
0925 07:30:16.384068 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:16.384070 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:16.384097 (+ 27us) server_negotiation.cc:407] Connection header received
0925 07:30:16.384133 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:16.384134 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:16.384140 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:16.384174 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:30:16.384371 (+ 197us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36516: BlockingRecv error: recv got EOF from 127.0.0.1:36516 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260925 07:30:16.384497 9480 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790321392278483-9480-0/krb5kdc/krb5cc' of type FILE
Sep 25 07:30:16 dist-test-slave-57fn krb5kdc[14994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321416, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:30:16.385382 9480 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 07:30:16 dist-test-slave-57fn krb5kdc[14994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321416, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:30:16.386278 9480 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 07:30:16 dist-test-slave-57fn krb5kdc[14994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321416, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.66.1@KRBTEST.COM
I20260925 07:30:16.392520 15505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:16.386755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36520 (local address 127.9.66.1:34967)
0925 07:30:16.386812 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:16.386818 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:16.386826 (+ 8us) server_negotiation.cc:407] Connection header received
0925 07:30:16.386841 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:16.386841 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:16.386847 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:16.386891 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:30:16.387207 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:16.387655 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:16.388277 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:16.388445 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:16.388492 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:16.390289 (+ 1797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:30:16.390290 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:30:16.390290 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:30:16.390296 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:30:16.391596 (+ 1300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:16.391896 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:16.391897 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:16.391897 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:16.391942 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:16.392107 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:16.392108 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:16.392108 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:16.392185 (+ 77us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:30:16.392255 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:16.392333 (+ 78us) server_negotiation.cc:299] Negotiation successful
0925 07:30:16.392380 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 07:30:16.393126 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 15101
I20260925 07:30:16.398504 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 15235
I20260925 07:30:16.403115 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 15369
I20260925 07:30:16.407114 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 15007
2026-09-25T07:30:16Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1862 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 07:30:16 dist-test-slave-57fn krb5kdc[15518](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 07:30:16 dist-test-slave-57fn krb5kdc[15518](info): set up 2 sockets
Sep 25 07:30:16 dist-test-slave-57fn krb5kdc[15518](info): commencing operation
krb5kdc: starting...
W20260925 07:30:17.469878 9480 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/krb5kdc/test-user.keytab.
Sep 25 07:30:17 dist-test-slave-57fn krb5kdc[15518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321417, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T07:30:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T07:30:17Z Disabled control of system clock
WARNING: no policy specified for oryx/127.9.66.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.66.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.66.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/krb5kdc/oryx_127.9.66.62.keytab.
Entry for principal oryx/127.9.66.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/krb5kdc/oryx_127.9.66.62.keytab.
I20260925 07:30:17.589033 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.66.62:34429
--webserver_interface=127.9.66.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/krb5kdc/oryx_127.9.66.62.keytab
--principal=oryx/127.9.66.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:39737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.66.62:34429
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:30:17.668450 15531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:30:17.668648 15531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:30:17.668670 15531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:30:17.669798 15531 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:30:17.669837 15531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:30:17.669852 15531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:30:17.669864 15531 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:30:17.669876 15531 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:30:17.671257 15531 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:39737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.66.62:34429
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/krb5kdc/oryx_127.9.66.62.keytab
--principal=oryx/127.9.66.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.66.62:34429
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.66.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15531
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:30:17.671438 15531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:30:17.671591 15531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:30:17.676208 15537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:30:17.676203 15539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:30:17.676204 15536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:30:17.676291 15531 server_base.cc:1034] running on GCE node
I20260925 07:30:17.676681 15531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:30:17.677001 15531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:30:17.678170 15531 hybrid_clock.cc:648] HybridClock initialized: now 1790321417678146 us; error 36 us; skew 500 ppm
Sep 25 07:30:17 dist-test-slave-57fn krb5kdc[15518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321417, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.66.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:30:17.680640 15531 init.cc:377] Logged in from keytab as oryx/127.9.66.62@KRBTEST.COM (short username oryx)
I20260925 07:30:17.681115 15531 webserver.cc:460] Webserver started at http://127.9.66.62:41247/ using document root <none> and password file <none>
I20260925 07:30:17.681331 15531 fs_manager.cc:359] Metadata directory not provided
I20260925 07:30:17.681396 15531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:30:17.681572 15531 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:30:17.682607 15531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/master-0/data/instance:
uuid: "438b938b95aa4027a1d63ae68aef56ad"
format_stamp: "Formatted at 2026-09-25 07:30:17 on dist-test-slave-57fn"
server_key: "6a35e65625b5c5bcdece9479c96dd380"
server_key_iv: "27cc5d5c067a56c027a24589139a7f1c"
server_key_version: "encryptionkey@0"
I20260925 07:30:17.682966 15531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/master-0/wal/instance:
uuid: "438b938b95aa4027a1d63ae68aef56ad"
format_stamp: "Formatted at 2026-09-25 07:30:17 on dist-test-slave-57fn"
server_key: "6a35e65625b5c5bcdece9479c96dd380"
server_key_iv: "27cc5d5c067a56c027a24589139a7f1c"
server_key_version: "encryptionkey@0"
I20260925 07:30:17.684363 15531 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 07:30:17.685142 15546 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:17.685321 15531 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 07:30:17.685418 15531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/master-0/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/master-0/wal
uuid: "438b938b95aa4027a1d63ae68aef56ad"
format_stamp: "Formatted at 2026-09-25 07:30:17 on dist-test-slave-57fn"
server_key: "6a35e65625b5c5bcdece9479c96dd380"
server_key_iv: "27cc5d5c067a56c027a24589139a7f1c"
server_key_version: "encryptionkey@0"
I20260925 07:30:17.685500 15531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:30:17.697115 15531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:30:17.699334 15531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:30:17.699493 15531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:30:17.703737 15531 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.62:34429
I20260925 07:30:17.703790 15598 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.62:34429 every 8 connection(s)
I20260925 07:30:17.704152 15531 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
I20260925 07:30:17.705132 15599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:30:17.707288 15599 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 438b938b95aa4027a1d63ae68aef56ad: Bootstrap starting.
I20260925 07:30:17.707791 15599 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 438b938b95aa4027a1d63ae68aef56ad: Neither blocks nor log segments found. Creating new log.
I20260925 07:30:17.708015 15599 log.cc:824] T 00000000000000000000000000000000 P 438b938b95aa4027a1d63ae68aef56ad: Log is configured to *not* fsync() on all Append() calls
I20260925 07:30:17.708653 15599 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 438b938b95aa4027a1d63ae68aef56ad: No bootstrap required, opened a new log
I20260925 07:30:17.710242 15599 raft_consensus.cc:348] T 00000000000000000000000000000000 P 438b938b95aa4027a1d63ae68aef56ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "438b938b95aa4027a1d63ae68aef56ad" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 34429 } }
I20260925 07:30:17.710369 15599 raft_consensus.cc:374] T 00000000000000000000000000000000 P 438b938b95aa4027a1d63ae68aef56ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:30:17.710390 15599 raft_consensus.cc:729] T 00000000000000000000000000000000 P 438b938b95aa4027a1d63ae68aef56ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 438b938b95aa4027a1d63ae68aef56ad, State: Initialized, Role: FOLLOWER
I20260925 07:30:17.710486 15599 consensus_queue.cc:260] T 00000000000000000000000000000000 P 438b938b95aa4027a1d63ae68aef56ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "438b938b95aa4027a1d63ae68aef56ad" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 34429 } }
I20260925 07:30:17.710536 15599 raft_consensus.cc:388] T 00000000000000000000000000000000 P 438b938b95aa4027a1d63ae68aef56ad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 07:30:17.710556 15599 raft_consensus.cc:482] T 00000000000000000000000000000000 P 438b938b95aa4027a1d63ae68aef56ad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 07:30:17.710632 15599 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 438b938b95aa4027a1d63ae68aef56ad [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:30:17.711376 15599 raft_consensus.cc:504] T 00000000000000000000000000000000 P 438b938b95aa4027a1d63ae68aef56ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "438b938b95aa4027a1d63ae68aef56ad" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 34429 } }
I20260925 07:30:17.711491 15599 leader_election.cc:302] T 00000000000000000000000000000000 P 438b938b95aa4027a1d63ae68aef56ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 438b938b95aa4027a1d63ae68aef56ad; no voters:
I20260925 07:30:17.711661 15599 leader_election.cc:288] T 00000000000000000000000000000000 P 438b938b95aa4027a1d63ae68aef56ad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 07:30:17.711699 15602 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 438b938b95aa4027a1d63ae68aef56ad [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:30:17.711920 15599 sys_catalog.cc:564] T 00000000000000000000000000000000 P 438b938b95aa4027a1d63ae68aef56ad [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:30:17.712026 15602 raft_consensus.cc:686] T 00000000000000000000000000000000 P 438b938b95aa4027a1d63ae68aef56ad [term 1 LEADER]: Becoming Leader. State: Replica: 438b938b95aa4027a1d63ae68aef56ad, State: Running, Role: LEADER
I20260925 07:30:17.712186 15602 consensus_queue.cc:237] T 00000000000000000000000000000000 P 438b938b95aa4027a1d63ae68aef56ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "438b938b95aa4027a1d63ae68aef56ad" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 34429 } }
I20260925 07:30:17.712602 15604 sys_catalog.cc:455] T 00000000000000000000000000000000 P 438b938b95aa4027a1d63ae68aef56ad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 438b938b95aa4027a1d63ae68aef56ad. Latest consensus state: current_term: 1 leader_uuid: "438b938b95aa4027a1d63ae68aef56ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "438b938b95aa4027a1d63ae68aef56ad" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 34429 } } }
I20260925 07:30:17.712705 15604 sys_catalog.cc:458] T 00000000000000000000000000000000 P 438b938b95aa4027a1d63ae68aef56ad [sys.catalog]: This master's current role is: LEADER
I20260925 07:30:17.712924 15611 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 07:30:17.713016 15603 sys_catalog.cc:455] T 00000000000000000000000000000000 P 438b938b95aa4027a1d63ae68aef56ad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "438b938b95aa4027a1d63ae68aef56ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "438b938b95aa4027a1d63ae68aef56ad" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 34429 } } }
I20260925 07:30:17.713096 15603 sys_catalog.cc:458] T 00000000000000000000000000000000 P 438b938b95aa4027a1d63ae68aef56ad [sys.catalog]: This master's current role is: LEADER
I20260925 07:30:17.713368 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 15531
I20260925 07:30:17.713398 15611 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 07:30:17.713485 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/master-0/wal/instance
I20260925 07:30:17.713719 9480 external_mini_cluster.cc:1428] Setting key 401fcc7c0f9fef96f4e4be53e347f9aa
I20260925 07:30:17.714970 15611 catalog_manager.cc:1347] Generated new cluster ID: 1f6a3af636aa4ddb853f1ed147c46d72
I20260925 07:30:17.715027 15611 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 07:30:17 dist-test-slave-57fn krb5kdc[15518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321417, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.66.62@KRBTEST.COM
I20260925 07:30:17.727536 15611 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 07:30:17.728175 15611 catalog_manager.cc:1504] Loading token signing keys...
I20260925 07:30:17.728790 15621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:17.714321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33784 (local address 127.9.66.62:34429)
0925 07:30:17.717671 (+ 3350us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:17.717680 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:17.717715 (+ 35us) server_negotiation.cc:407] Connection header received
0925 07:30:17.717828 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:17.717835 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:17.717888 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:17.718034 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:30:17.718877 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:17.720645 (+ 1768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:17.721655 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:17.722290 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:17.722398 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:17.724941 (+ 2543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:30:17.724945 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:30:17.724946 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:30:17.724957 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:30:17.726858 (+ 1901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:17.727259 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:17.727262 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:17.727262 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:17.727380 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:17.727631 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:17.727632 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:17.727633 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:17.727872 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:30:17.727995 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:17.728451 (+ 456us) server_negotiation.cc:299] Negotiation successful
0925 07:30:17.728582 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3307,"thread_start_us":72,"threads_started":1}
I20260925 07:30:17.734259 15611 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 438b938b95aa4027a1d63ae68aef56ad: Generated new TSK 0
I20260925 07:30:17.734436 15611 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.9.66.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.66.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.66.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/krb5kdc/oryx_127.9.66.1.keytab.
Entry for principal oryx/127.9.66.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/krb5kdc/oryx_127.9.66.1.keytab.
I20260925 07:30:17.772439 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.66.1:0
--local_ip_for_outbound_sockets=127.9.66.1
--webserver_interface=127.9.66.1
--webserver_port=0
--tserver_master_addrs=127.9.66.62:34429
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/krb5kdc/oryx_127.9.66.1.keytab
--principal=oryx/127.9.66.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:39737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:30:17.872082 15625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:30:17.872287 15625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:30:17.872311 15625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:30:17.873363 15625 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:30:17.873394 15625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:30:17.873420 15625 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.66.1
I20260925 07:30:17.874760 15625 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:39737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/krb5kdc/oryx_127.9.66.1.keytab
--principal=oryx/127.9.66.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.66.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.66.1
--webserver_port=0
--tserver_master_addrs=127.9.66.62:34429
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15625
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.66.1
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:30:17.874926 15625 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:30:17.875082 15625 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:30:17.879752 15633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:30:17.879870 15631 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:30:17.879817 15630 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:30:17.879875 15625 server_base.cc:1034] running on GCE node
I20260925 07:30:17.880115 15625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:30:17.880352 15625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:30:17.881551 15625 hybrid_clock.cc:648] HybridClock initialized: now 1790321417881533 us; error 30 us; skew 500 ppm
Sep 25 07:30:17 dist-test-slave-57fn krb5kdc[15518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321417, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.66.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:30:17.883838 15625 init.cc:377] Logged in from keytab as oryx/127.9.66.1@KRBTEST.COM (short username oryx)
I20260925 07:30:17.884294 15625 webserver.cc:460] Webserver started at http://127.9.66.1:44065/ using document root <none> and password file <none>
I20260925 07:30:17.884518 15625 fs_manager.cc:359] Metadata directory not provided
I20260925 07:30:17.884582 15625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:30:17.884706 15625 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:30:17.885764 15625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/ts-0/data/instance:
uuid: "0d7f890138dd42a885a27044b0cb5f9d"
format_stamp: "Formatted at 2026-09-25 07:30:17 on dist-test-slave-57fn"
server_key: "8026a49b5cc4befa3eafd2147c36ebc6"
server_key_iv: "b6d308c8841e8c12ee1f0e4ef4575155"
server_key_version: "encryptionkey@0"
I20260925 07:30:17.886111 15625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/ts-0/wal/instance:
uuid: "0d7f890138dd42a885a27044b0cb5f9d"
format_stamp: "Formatted at 2026-09-25 07:30:17 on dist-test-slave-57fn"
server_key: "8026a49b5cc4befa3eafd2147c36ebc6"
server_key_iv: "b6d308c8841e8c12ee1f0e4ef4575155"
server_key_version: "encryptionkey@0"
I20260925 07:30:17.887424 15625 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 07:30:17.888216 15640 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:17.888448 15625 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:30:17.888521 15625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/ts-0/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/ts-0/wal
uuid: "0d7f890138dd42a885a27044b0cb5f9d"
format_stamp: "Formatted at 2026-09-25 07:30:17 on dist-test-slave-57fn"
server_key: "8026a49b5cc4befa3eafd2147c36ebc6"
server_key_iv: "b6d308c8841e8c12ee1f0e4ef4575155"
server_key_version: "encryptionkey@0"
I20260925 07:30:17.888616 15625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:30:17.909405 15625 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:30:17.911207 15625 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:30:17.911366 15625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:30:17.911620 15625 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:30:17.911970 15625 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:30:17.912046 15625 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:17.912081 15625 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:30:17.912108 15625 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:17.918164 15625 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.1:36507
I20260925 07:30:17.918219 15753 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.1:36507 every 8 connection(s)
I20260925 07:30:17.918506 15625 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/ts-0/data/info.pb
Sep 25 07:30:17 dist-test-slave-57fn krb5kdc[15518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321417, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.66.1@KRBTEST.COM for oryx/127.9.66.62@KRBTEST.COM
I20260925 07:30:17.926738 15621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:17.919251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.1:56035 (local address 127.9.66.62:34429)
0925 07:30:17.919322 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:17.919325 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:17.919492 (+ 167us) server_negotiation.cc:407] Connection header received
0925 07:30:17.919991 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:17.919992 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:17.920002 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:17.920057 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:30:17.920920 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:17.921418 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:17.922372 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:17.922560 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:17.922614 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:17.924520 (+ 1906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:30:17.924521 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:30:17.924521 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:30:17.924527 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:30:17.925686 (+ 1159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:17.926015 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:17.926016 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:17.926016 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:17.926061 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:17.926266 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:17.926266 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:17.926267 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:17.926359 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:30:17.926405 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:17.926613 (+ 208us) server_negotiation.cc:299] Negotiation successful
0925 07:30:17.926677 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 07:30:17.926941 15756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:17.919223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34429 (local address 127.9.66.1:56035)
0925 07:30:17.919364 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:17.919403 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 07:30:17.919868 (+ 465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:30:17.920114 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:30:17.920117 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:30:17.920650 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:30:17.920841 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:17.920845 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:17.921497 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:17.921498 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:17.922295 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:17.922297 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:17.922635 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:17.922635 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:17.922744 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:17.923155 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:30:17.923162 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:30:17.924440 (+ 1278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:30:17.925750 (+ 1310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:30:17.925755 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:30:17.925757 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:30:17.925954 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:30:17.926123 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:30:17.926124 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:30:17.926124 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:30:17.926211 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:30:17.926411 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:30:17.926414 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:30:17.926513 (+ 99us) client_negotiation.cc:769] Sending connection context
0925 07:30:17.926587 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 07:30:17.926707 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":77,"threads_started":1}
I20260925 07:30:17.927515 15754 heartbeater.cc:347] Connected to a master server at 127.9.66.62:34429
I20260925 07:30:17.927601 15754 heartbeater.cc:464] Registering TS with master...
I20260925 07:30:17.927732 15754 heartbeater.cc:511] Master 127.9.66.62:34429 requested a full tablet report, sending...
I20260925 07:30:17.928015 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 15625
I20260925 07:30:17.928130 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/ts-0/wal/instance
I20260925 07:30:17.928330 9480 external_mini_cluster.cc:1428] Setting key aa0c8eb176ee94d01485f83e561cc1ec
I20260925 07:30:17.928331 15563 ts_manager.cc:194] Registered new tserver with Master: 0d7f890138dd42a885a27044b0cb5f9d (127.9.66.1:36507)
I20260925 07:30:17.929023 15563 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.66.1@KRBTEST.COM'} at 127.9.66.1:56035
WARNING: no policy specified for oryx/127.9.66.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.66.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.66.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/krb5kdc/oryx_127.9.66.2.keytab.
Entry for principal oryx/127.9.66.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/krb5kdc/oryx_127.9.66.2.keytab.
I20260925 07:30:17.951948 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.66.2:0
--local_ip_for_outbound_sockets=127.9.66.2
--webserver_interface=127.9.66.2
--webserver_port=0
--tserver_master_addrs=127.9.66.62:34429
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/krb5kdc/oryx_127.9.66.2.keytab
--principal=oryx/127.9.66.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:39737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:30:18.030489 15759 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:30:18.030766 15759 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:30:18.030809 15759 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:30:18.032071 15759 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:30:18.032116 15759 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:30:18.032145 15759 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.66.2
I20260925 07:30:18.033490 15759 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:39737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/krb5kdc/oryx_127.9.66.2.keytab
--principal=oryx/127.9.66.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.66.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.66.2
--webserver_port=0
--tserver_master_addrs=127.9.66.62:34429
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15759
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.66.2
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:30:18.033726 15759 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:30:18.033890 15759 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:30:18.039127 15767 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:30:18.039091 15765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:30:18.039091 15764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:30:18.039284 15759 server_base.cc:1034] running on GCE node
I20260925 07:30:18.039593 15759 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:30:18.039868 15759 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:30:18.041011 15759 hybrid_clock.cc:648] HybridClock initialized: now 1790321418041001 us; error 30 us; skew 500 ppm
Sep 25 07:30:18 dist-test-slave-57fn krb5kdc[15518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321418, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.66.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:30:18.043633 15759 init.cc:377] Logged in from keytab as oryx/127.9.66.2@KRBTEST.COM (short username oryx)
I20260925 07:30:18.044059 15759 webserver.cc:460] Webserver started at http://127.9.66.2:45989/ using document root <none> and password file <none>
I20260925 07:30:18.044279 15759 fs_manager.cc:359] Metadata directory not provided
I20260925 07:30:18.044323 15759 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:30:18.044477 15759 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:30:18.045496 15759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/ts-1/data/instance:
uuid: "a90537ab87b444eda9751362d5b6febb"
format_stamp: "Formatted at 2026-09-25 07:30:18 on dist-test-slave-57fn"
server_key: "8b58671dc72370857e113d54d0e6b316"
server_key_iv: "cb79ab6a97ac7bd92b6a5c09125f5001"
server_key_version: "encryptionkey@0"
I20260925 07:30:18.045935 15759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/ts-1/wal/instance:
uuid: "a90537ab87b444eda9751362d5b6febb"
format_stamp: "Formatted at 2026-09-25 07:30:18 on dist-test-slave-57fn"
server_key: "8b58671dc72370857e113d54d0e6b316"
server_key_iv: "cb79ab6a97ac7bd92b6a5c09125f5001"
server_key_version: "encryptionkey@0"
I20260925 07:30:18.047212 15759 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 07:30:18.047924 15774 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:18.048116 15759 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 07:30:18.048187 15759 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/ts-1/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/ts-1/wal
uuid: "a90537ab87b444eda9751362d5b6febb"
format_stamp: "Formatted at 2026-09-25 07:30:18 on dist-test-slave-57fn"
server_key: "8b58671dc72370857e113d54d0e6b316"
server_key_iv: "cb79ab6a97ac7bd92b6a5c09125f5001"
server_key_version: "encryptionkey@0"
I20260925 07:30:18.048274 15759 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:30:18.063498 15759 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:30:18.065881 15759 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:30:18.066069 15759 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:30:18.066304 15759 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:30:18.066648 15759 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:30:18.066708 15759 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:18.066751 15759 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:30:18.066776 15759 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:18.072640 15759 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.2:37737
I20260925 07:30:18.072685 15887 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.2:37737 every 8 connection(s)
I20260925 07:30:18.073024 15759 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/ts-1/data/info.pb
I20260925 07:30:18.076347 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 15759
I20260925 07:30:18.076472 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/ts-1/wal/instance
I20260925 07:30:18.076666 9480 external_mini_cluster.cc:1428] Setting key a1724d37ed095aaf543b177efacc993c
Sep 25 07:30:18 dist-test-slave-57fn krb5kdc[15518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321418, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.66.2@KRBTEST.COM for oryx/127.9.66.62@KRBTEST.COM
I20260925 07:30:18.084105 15621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:18.073749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.2:33489 (local address 127.9.66.62:34429)
0925 07:30:18.073797 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:18.073799 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:18.074021 (+ 222us) server_negotiation.cc:407] Connection header received
0925 07:30:18.074465 (+ 444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:18.074466 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:18.074476 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:18.074529 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:30:18.075383 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:18.075864 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:18.077065 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:18.077384 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:18.077453 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:18.081180 (+ 3727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:30:18.081181 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:30:18.081182 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:30:18.081193 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:30:18.082656 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:18.083154 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:18.083155 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:18.083155 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:18.083215 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:18.083494 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:18.083495 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:18.083495 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:18.083591 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:30:18.083639 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:18.083974 (+ 335us) server_negotiation.cc:299] Negotiation successful
0925 07:30:18.084041 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 07:30:18.084412 15890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:18.073769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34429 (local address 127.9.66.2:33489)
0925 07:30:18.073909 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:18.073941 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0925 07:30:18.074387 (+ 446us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:30:18.074580 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:30:18.074583 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:30:18.075124 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:30:18.075309 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:18.075314 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:18.075946 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:18.075946 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:18.076978 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:18.076981 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:18.078666 (+ 1685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:18.078667 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:18.078772 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:18.079311 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:30:18.079319 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:30:18.081081 (+ 1762us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:30:18.082746 (+ 1665us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:30:18.082752 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:30:18.082754 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:30:18.083072 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:30:18.083287 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:30:18.083288 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:30:18.083288 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:30:18.083424 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:30:18.083665 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:30:18.083670 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:30:18.083831 (+ 161us) client_negotiation.cc:769] Sending connection context
0925 07:30:18.083960 (+ 129us) client_negotiation.cc:240] Negotiation successful
0925 07:30:18.084129 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":88,"threads_started":1}
WARNING: no policy specified for oryx/127.9.66.3@KRBTEST.COM; defaulting to no policy
I20260925 07:30:18.084997 15888 heartbeater.cc:347] Connected to a master server at 127.9.66.62:34429
I20260925 07:30:18.085103 15888 heartbeater.cc:464] Registering TS with master...
I20260925 07:30:18.085278 15888 heartbeater.cc:511] Master 127.9.66.62:34429 requested a full tablet report, sending...
I20260925 07:30:18.085678 15563 ts_manager.cc:194] Registered new tserver with Master: a90537ab87b444eda9751362d5b6febb (127.9.66.2:37737)
I20260925 07:30:18.086086 15563 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.66.2@KRBTEST.COM'} at 127.9.66.2:33489
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.66.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.66.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/krb5kdc/oryx_127.9.66.3.keytab.
Entry for principal oryx/127.9.66.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/krb5kdc/oryx_127.9.66.3.keytab.
I20260925 07:30:18.104470 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.66.3:0
--local_ip_for_outbound_sockets=127.9.66.3
--webserver_interface=127.9.66.3
--webserver_port=0
--tserver_master_addrs=127.9.66.62:34429
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/krb5kdc/oryx_127.9.66.3.keytab
--principal=oryx/127.9.66.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:39737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:30:18.182909 15893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:30:18.183108 15893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:30:18.183131 15893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:30:18.184294 15893 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:30:18.184338 15893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:30:18.184365 15893 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.66.3
I20260925 07:30:18.185814 15893 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:39737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/krb5kdc/oryx_127.9.66.3.keytab
--principal=oryx/127.9.66.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.66.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.66.3
--webserver_port=0
--tserver_master_addrs=127.9.66.62:34429
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15893
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.66.3
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:30:18.186044 15893 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:30:18.186285 15893 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:30:18.191510 15899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:30:18.191541 15901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:30:18.191551 15893 server_base.cc:1034] running on GCE node
W20260925 07:30:18.191521 15898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:30:18.191987 15893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:30:18.192248 15893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:30:18.193436 15893 hybrid_clock.cc:648] HybridClock initialized: now 1790321418193433 us; error 37 us; skew 500 ppm
Sep 25 07:30:18 dist-test-slave-57fn krb5kdc[15518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321418, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.66.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:30:18.196038 15893 init.cc:377] Logged in from keytab as oryx/127.9.66.3@KRBTEST.COM (short username oryx)
I20260925 07:30:18.196465 15893 webserver.cc:460] Webserver started at http://127.9.66.3:45871/ using document root <none> and password file <none>
I20260925 07:30:18.196693 15893 fs_manager.cc:359] Metadata directory not provided
I20260925 07:30:18.196764 15893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:30:18.196918 15893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:30:18.198020 15893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/ts-2/data/instance:
uuid: "9bfdbd8c5f3347a5a98a38d05d3fb756"
format_stamp: "Formatted at 2026-09-25 07:30:18 on dist-test-slave-57fn"
server_key: "7e55cb8e25bf6337372e1286fd7f7e52"
server_key_iv: "b583798a1b7468fbb2eda9481831d4c6"
server_key_version: "encryptionkey@0"
I20260925 07:30:18.198410 15893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/ts-2/wal/instance:
uuid: "9bfdbd8c5f3347a5a98a38d05d3fb756"
format_stamp: "Formatted at 2026-09-25 07:30:18 on dist-test-slave-57fn"
server_key: "7e55cb8e25bf6337372e1286fd7f7e52"
server_key_iv: "b583798a1b7468fbb2eda9481831d4c6"
server_key_version: "encryptionkey@0"
I20260925 07:30:18.199743 15893 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 07:30:18.200613 15908 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:18.200829 15893 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:30:18.200945 15893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/ts-2/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/ts-2/wal
uuid: "9bfdbd8c5f3347a5a98a38d05d3fb756"
format_stamp: "Formatted at 2026-09-25 07:30:18 on dist-test-slave-57fn"
server_key: "7e55cb8e25bf6337372e1286fd7f7e52"
server_key_iv: "b583798a1b7468fbb2eda9481831d4c6"
server_key_version: "encryptionkey@0"
I20260925 07:30:18.201040 15893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:30:18.217407 15893 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:30:18.219372 15893 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:30:18.219534 15893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:30:18.219799 15893 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:30:18.220108 15893 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:30:18.220139 15893 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:18.220203 15893 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:30:18.220232 15893 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:18.225757 15893 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.3:34477
I20260925 07:30:18.225837 16021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.3:34477 every 8 connection(s)
I20260925 07:30:18.226187 15893 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/ts-2/data/info.pb
I20260925 07:30:18.228796 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 15893
I20260925 07:30:18.228907 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/ts-2/wal/instance
I20260925 07:30:18.229070 9480 external_mini_cluster.cc:1428] Setting key 547fe1a40f95491d1d0438acd7555478
Sep 25 07:30:18 dist-test-slave-57fn krb5kdc[15518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321418, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.66.3@KRBTEST.COM for oryx/127.9.66.62@KRBTEST.COM
I20260925 07:30:18.235406 16024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:18.226860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34429 (local address 127.9.66.3:33835)
0925 07:30:18.227048 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:18.227082 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 07:30:18.227551 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:30:18.227805 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:30:18.227808 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:30:18.228368 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:30:18.228549 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:18.228553 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:18.229397 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:18.229399 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:18.230443 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:18.230446 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:18.230771 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:18.230772 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:18.230866 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:18.231308 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:30:18.231316 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:30:18.232708 (+ 1392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:30:18.234193 (+ 1485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:30:18.234199 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:30:18.234201 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:30:18.234458 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:30:18.234630 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:30:18.234631 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:30:18.234631 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:30:18.234747 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:30:18.234963 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:30:18.234967 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:30:18.235069 (+ 102us) client_negotiation.cc:769] Sending connection context
0925 07:30:18.235147 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 07:30:18.235258 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":95,"threads_started":1}
I20260925 07:30:18.235425 15621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:18.226910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.3:33835 (local address 127.9.66.62:34429)
0925 07:30:18.227011 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:18.227017 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:18.227162 (+ 145us) server_negotiation.cc:407] Connection header received
0925 07:30:18.227633 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:18.227634 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:18.227645 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:18.227731 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:30:18.228636 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:18.229285 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:18.230511 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:18.230692 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:18.230745 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:18.232786 (+ 2041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:30:18.232787 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:30:18.232787 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:30:18.232793 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:30:18.234115 (+ 1322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:18.234522 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:18.234523 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:18.234523 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:18.234571 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:18.234805 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:18.234806 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:18.234806 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:18.234907 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:30:18.234959 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:18.235247 (+ 288us) server_negotiation.cc:299] Negotiation successful
0925 07:30:18.235333 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 07:30:18.235913 16022 heartbeater.cc:347] Connected to a master server at 127.9.66.62:34429
I20260925 07:30:18.236020 16022 heartbeater.cc:464] Registering TS with master...
I20260925 07:30:18.236207 16022 heartbeater.cc:511] Master 127.9.66.62:34429 requested a full tablet report, sending...
I20260925 07:30:18.236644 15563 ts_manager.cc:194] Registered new tserver with Master: 9bfdbd8c5f3347a5a98a38d05d3fb756 (127.9.66.3:34477)
I20260925 07:30:18.237005 15563 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.66.3@KRBTEST.COM'} at 127.9.66.3:33835
I20260925 07:30:18.240814 9480 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.9.66.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.66.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.66.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/krb5kdc/oryx_127.9.66.61.keytab.
Entry for principal oryx/127.9.66.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/krb5kdc/oryx_127.9.66.61.keytab.
I20260925 07:30:18.264830 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.66.61:33901
--webserver_interface=127.9.66.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/krb5kdc/oryx_127.9.66.61.keytab
--principal=oryx/127.9.66.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:39737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.66.62:34429,127.9.66.61:33901
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:30:18.345387 16027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:30:18.345642 16027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:30:18.345669 16027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:30:18.346747 16027 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:30:18.346778 16027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:30:18.346792 16027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:30:18.346804 16027 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:30:18.346817 16027 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:30:18.348179 16027 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:39737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.9.66.62:34429,127.9.66.61:33901
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/krb5kdc/oryx_127.9.66.61.keytab
--principal=oryx/127.9.66.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.66.61:33901
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.9.66.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16027
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
W20260925 07:30:18.348280 16027 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.9.66.62:34429,127.9.66.61:33901'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260925 07:30:18.348381 16027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:30:18.348538 16027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:30:18.353160 16033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:30:18.353163 16032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:30:18.353281 16027 server_base.cc:1034] running on GCE node
W20260925 07:30:18.353204 16035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:30:18.353739 16027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:30:18.354059 16027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:30:18.355269 16027 hybrid_clock.cc:648] HybridClock initialized: now 1790321418355246 us; error 33 us; skew 500 ppm
Sep 25 07:30:18 dist-test-slave-57fn krb5kdc[15518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321418, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.66.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:30:18.357653 16027 init.cc:377] Logged in from keytab as oryx/127.9.66.61@KRBTEST.COM (short username oryx)
I20260925 07:30:18.358018 16027 webserver.cc:460] Webserver started at http://127.9.66.61:45167/ using document root <none> and password file <none>
I20260925 07:30:18.358191 16027 fs_manager.cc:359] Metadata directory not provided
I20260925 07:30:18.358229 16027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:30:18.358309 16027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:30:18.359228 16027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/master-1/data/instance:
uuid: "01d8c4dcbc1244b28cfa2b85723a0a54"
format_stamp: "Formatted at 2026-09-25 07:30:18 on dist-test-slave-57fn"
server_key: "67d1b7ea23ef0237a230ffd26fe3673a"
server_key_iv: "56f70dccc10dd66e20706a0624a62c88"
server_key_version: "encryptionkey@0"
I20260925 07:30:18.359503 16027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/master-1/wal/instance:
uuid: "01d8c4dcbc1244b28cfa2b85723a0a54"
format_stamp: "Formatted at 2026-09-25 07:30:18 on dist-test-slave-57fn"
server_key: "67d1b7ea23ef0237a230ffd26fe3673a"
server_key_iv: "56f70dccc10dd66e20706a0624a62c88"
server_key_version: "encryptionkey@0"
I20260925 07:30:18.360687 16027 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 07:30:18.361431 16042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:18.361665 16027 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 07:30:18.361733 16027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/master-1/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/master-1/wal
uuid: "01d8c4dcbc1244b28cfa2b85723a0a54"
format_stamp: "Formatted at 2026-09-25 07:30:18 on dist-test-slave-57fn"
server_key: "67d1b7ea23ef0237a230ffd26fe3673a"
server_key_iv: "56f70dccc10dd66e20706a0624a62c88"
server_key_version: "encryptionkey@0"
I20260925 07:30:18.361781 16027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:30:18.378504 16027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:30:18.380416 16027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:30:18.380514 16027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:30:18.384363 16027 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.61:33901
I20260925 07:30:18.384436 16094 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.61:33901 every 8 connection(s)
I20260925 07:30:18.384765 16027 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/master-1/data/info.pb
I20260925 07:30:18.385754 16095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:30:18.387180 16095 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.66.62" port: 34429 } has no permanent_uuid. Determining permanent_uuid...
I20260925 07:30:18.389643 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 16027
Sep 25 07:30:18 dist-test-slave-57fn krb5kdc[15518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321418, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.66.61@KRBTEST.COM for oryx/127.9.66.62@KRBTEST.COM
Sep 25 07:30:18 dist-test-slave-57fn krb5kdc[15518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321417, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.66.61@KRBTEST.COM
I20260925 07:30:18.396792 15621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:18.387973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33798 (local address 127.9.66.62:34429)
0925 07:30:18.388037 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:18.388038 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:18.388268 (+ 230us) server_negotiation.cc:407] Connection header received
0925 07:30:18.388755 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:18.388756 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:18.388768 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:18.388832 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:30:18.389434 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:18.390059 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:18.391155 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:18.391427 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:18.391497 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:18.393925 (+ 2428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:30:18.393926 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:30:18.393927 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:30:18.393932 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:30:18.395341 (+ 1409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:18.395782 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:18.395783 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:18.395784 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:18.395859 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:18.396094 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:18.396095 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:18.396095 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:18.396284 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:30:18.396360 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:18.396613 (+ 253us) server_negotiation.cc:299] Negotiation successful
0925 07:30:18.396701 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 07:30:18.396940 16097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:18.387912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34429 (local address 127.0.0.1:33798)
0925 07:30:18.388133 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:18.388176 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 07:30:18.388639 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:30:18.388897 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:30:18.388900 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:30:18.389163 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:30:18.389339 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:18.389343 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:18.390163 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:18.390164 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:18.391074 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:18.391077 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:18.391511 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:18.391512 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:18.391643 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:18.392175 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:30:18.392184 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:30:18.393819 (+ 1635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:30:18.395423 (+ 1604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:30:18.395429 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:30:18.395431 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:30:18.395697 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:30:18.395934 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:30:18.395935 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:30:18.395935 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:30:18.396026 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:30:18.396360 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:30:18.396364 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:30:18.396467 (+ 103us) client_negotiation.cc:769] Sending connection context
0925 07:30:18.396547 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 07:30:18.396648 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":135,"threads_started":1}
I20260925 07:30:18.397543 16095 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.66.61" port: 33901 } has no permanent_uuid. Determining permanent_uuid...
I20260925 07:30:18.399645 16099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:18.390105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32902 (local address 127.9.66.61:33901)
0925 07:30:18.391206 (+ 1101us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:18.391210 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:18.391226 (+ 16us) server_negotiation.cc:407] Connection header received
0925 07:30:18.391242 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:18.391244 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:18.391259 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:18.391319 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:30:18.391952 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:18.393007 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:18.393882 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:18.394196 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:18.394280 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:18.396711 (+ 2431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:30:18.396717 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:30:18.396718 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:30:18.396724 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:30:18.398571 (+ 1847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:18.398902 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:18.398917 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:18.398918 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:18.398979 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:18.399187 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:18.399187 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:18.399188 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:18.399304 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:30:18.399410 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:18.399528 (+ 118us) server_negotiation.cc:299] Negotiation successful
0925 07:30:18.399586 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1078,"thread_start_us":72,"threads_started":1}
Sep 25 07:30:18 dist-test-slave-57fn krb5kdc[15518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321418, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.66.61@KRBTEST.COM for oryx/127.9.66.61@KRBTEST.COM
I20260925 07:30:18.404455 16097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:18.397783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.61:33901 (local address 127.0.0.1:32912)
0925 07:30:18.397821 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:18.397831 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 07:30:18.397871 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:30:18.398106 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:30:18.398106 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:30:18.398219 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:30:18.398261 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:18.398262 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:18.399098 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:18.399099 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:18.399841 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:18.399844 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:18.400440 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:18.400441 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:18.400499 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:18.400780 (+ 281us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:30:18.400783 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:30:18.401855 (+ 1072us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:30:18.403608 (+ 1753us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:30:18.403609 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:30:18.403609 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:30:18.403853 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:30:18.404007 (+ 154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:30:18.404008 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:30:18.404008 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:30:18.404057 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:30:18.404232 (+ 175us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:30:18.404232 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:30:18.404262 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 07:30:18.404334 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 07:30:18.404401 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20}
I20260925 07:30:18.404565 16100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:18.397841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32912 (local address 127.9.66.61:33901)
0925 07:30:18.397982 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:18.397984 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:18.397992 (+ 8us) server_negotiation.cc:407] Connection header received
0925 07:30:18.398012 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:18.398013 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:18.398021 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:18.398097 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:30:18.398347 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:18.399030 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:18.400153 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:18.400370 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:18.400441 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:18.401928 (+ 1487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:30:18.401929 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:30:18.401929 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:30:18.401934 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:30:18.403543 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:18.403905 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:18.403906 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:18.403906 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:18.403951 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:18.404097 (+ 146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:18.404098 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:18.404098 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:18.404177 (+ 79us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:30:18.404228 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:18.404351 (+ 123us) server_negotiation.cc:299] Negotiation successful
0925 07:30:18.404446 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":39,"server-negotiator.queue_time_us":120,"thread_start_us":71,"threads_started":1}
I20260925 07:30:18.405794 16095 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01d8c4dcbc1244b28cfa2b85723a0a54: Bootstrap starting.
I20260925 07:30:18.406394 16095 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 01d8c4dcbc1244b28cfa2b85723a0a54: Neither blocks nor log segments found. Creating new log.
I20260925 07:30:18.406675 16095 log.cc:824] T 00000000000000000000000000000000 P 01d8c4dcbc1244b28cfa2b85723a0a54: Log is configured to *not* fsync() on all Append() calls
I20260925 07:30:18.407318 16095 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01d8c4dcbc1244b28cfa2b85723a0a54: No bootstrap required, opened a new log
I20260925 07:30:18.408643 16095 raft_consensus.cc:348] T 00000000000000000000000000000000 P 01d8c4dcbc1244b28cfa2b85723a0a54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "438b938b95aa4027a1d63ae68aef56ad" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 34429 } } peers { permanent_uuid: "01d8c4dcbc1244b28cfa2b85723a0a54" member_type: VOTER last_known_addr { host: "127.9.66.61" port: 33901 } }
I20260925 07:30:18.408771 16095 raft_consensus.cc:374] T 00000000000000000000000000000000 P 01d8c4dcbc1244b28cfa2b85723a0a54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:30:18.408820 16095 raft_consensus.cc:729] T 00000000000000000000000000000000 P 01d8c4dcbc1244b28cfa2b85723a0a54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01d8c4dcbc1244b28cfa2b85723a0a54, State: Initialized, Role: FOLLOWER
I20260925 07:30:18.408941 16095 consensus_queue.cc:260] T 00000000000000000000000000000000 P 01d8c4dcbc1244b28cfa2b85723a0a54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "438b938b95aa4027a1d63ae68aef56ad" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 34429 } } peers { permanent_uuid: "01d8c4dcbc1244b28cfa2b85723a0a54" member_type: VOTER last_known_addr { host: "127.9.66.61" port: 33901 } }
I20260925 07:30:18.409204 16102 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01d8c4dcbc1244b28cfa2b85723a0a54 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "438b938b95aa4027a1d63ae68aef56ad" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 34429 } } peers { permanent_uuid: "01d8c4dcbc1244b28cfa2b85723a0a54" member_type: VOTER last_known_addr { host: "127.9.66.61" port: 33901 } } }
I20260925 07:30:18.409286 16102 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01d8c4dcbc1244b28cfa2b85723a0a54 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 07:30:18.409248 16095 sys_catalog.cc:564] T 00000000000000000000000000000000 P 01d8c4dcbc1244b28cfa2b85723a0a54 [sys.catalog]: configured and running, proceeding with master startup.
W20260925 07:30:18.410873 16113 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 01d8c4dcbc1244b28cfa2b85723a0a54: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 07:30:18.410960 16113 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 07:30:18.412441 16027 master_runner.cc:428] Detected that this master 01d8c4dcbc1244b28cfa2b85723a0a54 is joining an existing cluster
I20260925 07:30:18.412487 16027 master_runner.cc:432] Initiating AddMaster RPC to add 127.9.66.61:33901
I20260925 07:30:18.419899 16119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:18.413448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.61:33901 (local address 127.0.0.1:32920)
0925 07:30:18.413717 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:18.413727 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 07:30:18.413784 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:30:18.413893 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:30:18.413894 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:30:18.414107 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:30:18.414165 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:18.414166 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:18.415051 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:18.415052 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:18.415964 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:18.415966 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:18.416481 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:18.416481 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:18.416538 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:18.416895 (+ 357us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:30:18.416900 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:30:18.417252 (+ 352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:30:18.418794 (+ 1542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:30:18.418795 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:30:18.418796 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:30:18.419041 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:30:18.419247 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:30:18.419247 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:30:18.419247 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:30:18.419311 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:30:18.419568 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:30:18.419569 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:30:18.419612 (+ 43us) client_negotiation.cc:769] Sending connection context
0925 07:30:18.419697 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 07:30:18.419798 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"mutex_wait_us":19,"thread_start_us":117,"threads_started":1}
I20260925 07:30:18.419961 15621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:18.413681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33810 (local address 127.9.66.62:34429)
0925 07:30:18.413722 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:18.413723 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:18.413731 (+ 8us) server_negotiation.cc:407] Connection header received
0925 07:30:18.413751 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:18.413751 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:18.413759 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:18.413823 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:30:18.414230 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:18.414960 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:18.415744 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:18.415943 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:18.416029 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:18.417074 (+ 1045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:30:18.417075 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:30:18.417076 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:30:18.417082 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:30:18.418648 (+ 1566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:18.419102 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:18.419103 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:18.419104 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:18.419172 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:18.419435 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:18.419436 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:18.419436 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:18.419555 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:30:18.419613 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:18.419787 (+ 174us) server_negotiation.cc:299] Negotiation successful
0925 07:30:18.419859 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 07:30:18.420121 16100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:18.413480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32920 (local address 127.9.66.61:33901)
0925 07:30:18.413555 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:18.413557 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:18.413771 (+ 214us) server_negotiation.cc:407] Connection header received
0925 07:30:18.413834 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:18.413834 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:18.413844 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:18.413888 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:30:18.414264 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:18.414959 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:18.416059 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:18.416378 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:18.416418 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:18.417351 (+ 933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:30:18.417352 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:30:18.417352 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:30:18.417359 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:30:18.418679 (+ 1320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:18.419112 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:18.419113 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:18.419113 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:18.419171 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:18.419381 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:18.419382 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:18.419382 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:18.419497 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:30:18.419559 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:18.419745 (+ 186us) server_negotiation.cc:299] Negotiation successful
0925 07:30:18.419965 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":17}
I20260925 07:30:18.420408 16120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:18.413436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34429 (local address 127.0.0.1:33810)
0925 07:30:18.413629 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:18.413638 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 07:30:18.413692 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:30:18.413912 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:30:18.413917 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:30:18.414088 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:30:18.414140 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:18.414141 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:18.415044 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:18.415045 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:18.415654 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:18.415656 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:18.416048 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:18.416049 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:18.416107 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:18.416584 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:30:18.416590 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:30:18.416974 (+ 384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:30:18.418740 (+ 1766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:30:18.418741 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:30:18.418741 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:30:18.418953 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:30:18.419250 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:30:18.419251 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:30:18.419252 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:30:18.419338 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:30:18.419605 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:30:18.419606 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:30:18.419659 (+ 53us) client_negotiation.cc:769] Sending connection context
0925 07:30:18.420029 (+ 370us) client_negotiation.cc:240] Negotiation successful
0925 07:30:18.420122 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"mutex_wait_us":19,"thread_start_us":162,"threads_started":1}
Sep 25 07:30:18 dist-test-slave-57fn krb5kdc[15518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321417, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.66.62@KRBTEST.COM for oryx/127.9.66.61@KRBTEST.COM
I20260925 07:30:18.429606 16121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:18.422544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.61:33901 (local address 127.0.0.1:32926)
0925 07:30:18.422688 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:18.422724 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 07:30:18.422792 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:30:18.422917 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:30:18.422920 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:30:18.423247 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:30:18.423301 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:18.423303 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:18.423870 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:18.423871 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:18.424776 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:18.424778 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:18.425083 (+ 305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:18.425084 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:18.425174 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:18.425567 (+ 393us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:30:18.425572 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:30:18.426977 (+ 1405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:30:18.428612 (+ 1635us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:30:18.428617 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:30:18.428619 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:30:18.428807 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:30:18.429003 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:30:18.429003 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:30:18.429003 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:30:18.429052 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:30:18.429301 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:30:18.429306 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:30:18.429342 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 07:30:18.429403 (+ 61us) client_negotiation.cc:240] Negotiation successful
0925 07:30:18.429481 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":58,"threads_started":1}
I20260925 07:30:18.429631 16100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:18.422579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32926 (local address 127.9.66.61:33901)
0925 07:30:18.422610 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:18.422612 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:18.422786 (+ 174us) server_negotiation.cc:407] Connection header received
0925 07:30:18.422860 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:18.422860 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:18.422867 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:18.422911 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:30:18.423372 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:18.423792 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:18.424854 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:18.425012 (+ 158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:18.425056 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:18.427062 (+ 2006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:30:18.427063 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:30:18.427063 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:30:18.427069 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:30:18.428533 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:18.428880 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:18.428881 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:18.428881 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:18.428937 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:18.429114 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:18.429114 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:18.429115 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:18.429240 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:30:18.429304 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:18.429454 (+ 150us) server_negotiation.cc:299] Negotiation successful
0925 07:30:18.429549 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 07:30:18.430419 15563 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.9.66.61:33901: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "01d8c4dcbc1244b28cfa2b85723a0a54"
member_type: NON_VOTER
last_known_addr {
host: "127.9.66.61"
port: 33901
}
attrs {
promote: true
}
}
dest_uuid: "438b938b95aa4027a1d63ae68aef56ad"
cas_config_opid_index: -1
I20260925 07:30:18.430712 15563 consensus_queue.cc:237] T 00000000000000000000000000000000 P 438b938b95aa4027a1d63ae68aef56ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "438b938b95aa4027a1d63ae68aef56ad" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 34429 } } peers { permanent_uuid: "01d8c4dcbc1244b28cfa2b85723a0a54" member_type: NON_VOTER last_known_addr { host: "127.9.66.61" port: 33901 } attrs { promote: true } }
I20260925 07:30:18.431190 16123 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 438b938b95aa4027a1d63ae68aef56ad [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 01d8c4dcbc1244b28cfa2b85723a0a54 (127.9.66.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "438b938b95aa4027a1d63ae68aef56ad" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 34429 } } peers { permanent_uuid: "01d8c4dcbc1244b28cfa2b85723a0a54" member_type: NON_VOTER last_known_addr { host: "127.9.66.61" port: 33901 } attrs { promote: true } } }
I20260925 07:30:18.431417 16069 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 01d8c4dcbc1244b28cfa2b85723a0a54 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:30:18.432140 16123 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.9.66.61:33901
I20260925 07:30:18.432149 16069 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 01d8c4dcbc1244b28cfa2b85723a0a54 [term 1 FOLLOWER]: Refusing update from remote peer 438b938b95aa4027a1d63ae68aef56ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260925 07:30:18.432367 16122 sys_catalog.cc:455] T 00000000000000000000000000000000 P 438b938b95aa4027a1d63ae68aef56ad [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "438b938b95aa4027a1d63ae68aef56ad" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "438b938b95aa4027a1d63ae68aef56ad" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 34429 } } peers { permanent_uuid: "01d8c4dcbc1244b28cfa2b85723a0a54" member_type: NON_VOTER last_known_addr { host: "127.9.66.61" port: 33901 } attrs { promote: true } } }
I20260925 07:30:18.432426 16027 master.cc:537] Master@127.9.66.61:33901 shutting down...
I20260925 07:30:18.432438 16123 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 438b938b95aa4027a1d63ae68aef56ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "01d8c4dcbc1244b28cfa2b85723a0a54" member_type: NON_VOTER last_known_addr { host: "127.9.66.61" port: 33901 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:30:18.432437 16122 sys_catalog.cc:458] T 00000000000000000000000000000000 P 438b938b95aa4027a1d63ae68aef56ad [sys.catalog]: This master's current role is: LEADER
W20260925 07:30:18.433144 15550 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260925 07:30:18.434085 15550 consensus_peers.cc:487] T 00000000000000000000000000000000 P 438b938b95aa4027a1d63ae68aef56ad -> Peer 01d8c4dcbc1244b28cfa2b85723a0a54 (127.9.66.61:33901): Couldn't send request to peer 01d8c4dcbc1244b28cfa2b85723a0a54. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260925 07:30:18.435098 16027 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 01d8c4dcbc1244b28cfa2b85723a0a54 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 07:30:18.435199 16027 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 01d8c4dcbc1244b28cfa2b85723a0a54 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:30:18.435248 16027 tablet_replica.cc:330] T 00000000000000000000000000000000 P 01d8c4dcbc1244b28cfa2b85723a0a54: stopping tablet replica
I20260925 07:30:18.447746 16027 master.cc:552] Master@127.9.66.61:33901 shutdown complete.
I20260925 07:30:18.447799 16027 master_runner.cc:305] Clearing existing system tablet
I20260925 07:30:18.447926 16027 fs_manager.cc:359] Metadata directory not provided
I20260925 07:30:18.447974 16027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:30:18.448680 16027 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260925 07:30:18.449191 16127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:18.449337 16027 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:30:18.449396 16027 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 07:30:18.449414 16027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/master-1/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/master-1/wal
uuid: "01d8c4dcbc1244b28cfa2b85723a0a54"
format_stamp: "Formatted at 2026-09-25 07:30:18 on dist-test-slave-57fn"
server_key: "67d1b7ea23ef0237a230ffd26fe3673a"
server_key_iv: "56f70dccc10dd66e20706a0624a62c88"
server_key_version: "encryptionkey@0"
I20260925 07:30:18.450054 16027 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 01d8c4dcbc1244b28cfa2b85723a0a54: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 07:30:18.451900 16027 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 01d8c4dcbc1244b28cfa2b85723a0a54: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260925 07:30:18.451974 16027 log.cc:1196] T 00000000000000000000000000000000 P 01d8c4dcbc1244b28cfa2b85723a0a54: Deleting WAL directory at /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260925 07:30:18.452163 16027 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 01d8c4dcbc1244b28cfa2b85723a0a54: Deleting consensus metadata
I20260925 07:30:18.452294 16027 master_runner.cc:315] Copying system tablet from 127.9.66.62:34429
I20260925 07:30:18.452829 16027 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 01d8c4dcbc1244b28cfa2b85723a0a54: tablet copy: Beginning tablet copy session from remote peer at address 127.9.66.62:34429
I20260925 07:30:18.459064 15621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:18.453167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33818 (local address 127.9.66.62:34429)
0925 07:30:18.453213 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:18.453216 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:18.453312 (+ 96us) server_negotiation.cc:407] Connection header received
0925 07:30:18.453401 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:18.453402 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:18.453412 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:18.453469 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:30:18.454049 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:18.454528 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:18.455239 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:18.455433 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:18.455516 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:18.456477 (+ 961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:30:18.456478 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:30:18.456478 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:30:18.456485 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:30:18.457935 (+ 1450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:18.458284 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:18.458285 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:18.458285 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:18.458357 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:18.458547 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:18.458548 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:18.458549 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:18.458654 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:30:18.458753 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:18.458908 (+ 155us) server_negotiation.cc:299] Negotiation successful
0925 07:30:18.458977 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 07:30:18.459075 16133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:18.453106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34429 (local address 127.0.0.1:33818)
0925 07:30:18.453230 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:18.453240 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 07:30:18.453338 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:30:18.453474 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:30:18.453475 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:30:18.453666 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:30:18.453944 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:18.453946 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:18.454617 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:18.454618 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:18.455134 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:18.455136 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:18.455517 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:18.455518 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:18.455590 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:18.456057 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:30:18.456062 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:30:18.456339 (+ 277us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:30:18.458032 (+ 1693us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:30:18.458033 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:30:18.458033 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:30:18.458209 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:30:18.458423 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:30:18.458424 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:30:18.458424 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:30:18.458477 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:30:18.458746 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:30:18.458747 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:30:18.458789 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 07:30:18.458884 (+ 95us) client_negotiation.cc:240] Negotiation successful
0925 07:30:18.458979 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":71,"threads_started":1}
I20260925 07:30:18.459594 15583 tablet_copy_service.cc:140] P 438b938b95aa4027a1d63ae68aef56ad: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 01d8c4dcbc1244b28cfa2b85723a0a54 ({username='oryx', principal='oryx/127.9.66.61@KRBTEST.COM'} at 127.0.0.1:33818)
I20260925 07:30:18.459692 15583 tablet_copy_service.cc:161] P 438b938b95aa4027a1d63ae68aef56ad: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 01d8c4dcbc1244b28cfa2b85723a0a54 at {username='oryx', principal='oryx/127.9.66.61@KRBTEST.COM'} at 127.0.0.1:33818: session id = 01d8c4dcbc1244b28cfa2b85723a0a54-00000000000000000000000000000000
I20260925 07:30:18.460505 15583 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 438b938b95aa4027a1d63ae68aef56ad: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 07:30:18.461171 16027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260925 07:30:18.462054 16027 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 01d8c4dcbc1244b28cfa2b85723a0a54: tablet copy: Starting download of 0 data blocks...
I20260925 07:30:18.462239 16027 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 01d8c4dcbc1244b28cfa2b85723a0a54: tablet copy: Starting download of 1 WAL segments...
I20260925 07:30:18.463172 16027 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 01d8c4dcbc1244b28cfa2b85723a0a54: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 07:30:18.463900 15583 tablet_copy_service.cc:342] P 438b938b95aa4027a1d63ae68aef56ad: Request end of tablet copy session 01d8c4dcbc1244b28cfa2b85723a0a54-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.9.66.61@KRBTEST.COM'} at 127.0.0.1:33818
I20260925 07:30:18.463965 15583 tablet_copy_service.cc:433] P 438b938b95aa4027a1d63ae68aef56ad: ending tablet copy session 01d8c4dcbc1244b28cfa2b85723a0a54-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 01d8c4dcbc1244b28cfa2b85723a0a54
I20260925 07:30:18.466092 16027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:30:18.467242 16135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:30:18.467306 16134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:30:18.467315 16137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:30:18.469677 16027 server_base.cc:1034] running on GCE node
I20260925 07:30:18.469780 16027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:30:18.469909 16027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:30:18.471043 16027 hybrid_clock.cc:648] HybridClock initialized: now 1790321418471026 us; error 35 us; skew 500 ppm
Sep 25 07:30:18 dist-test-slave-57fn krb5kdc[15518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321418, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.66.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:30:18.476512 16027 init.cc:377] Logged in from keytab as oryx/127.9.66.61@KRBTEST.COM (short username oryx)
I20260925 07:30:18.476806 16027 webserver.cc:460] Webserver started at http://127.9.66.61:33489/ using document root <none> and password file <none>
I20260925 07:30:18.476923 16027 fs_manager.cc:359] Metadata directory not provided
I20260925 07:30:18.476951 16027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:30:18.477548 16027 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260925 07:30:18.478058 16144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:18.478191 16027 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 07:30:18.478241 16027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/master-1/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/master-1/wal
uuid: "01d8c4dcbc1244b28cfa2b85723a0a54"
format_stamp: "Formatted at 2026-09-25 07:30:18 on dist-test-slave-57fn"
server_key: "67d1b7ea23ef0237a230ffd26fe3673a"
server_key_iv: "56f70dccc10dd66e20706a0624a62c88"
server_key_version: "encryptionkey@0"
I20260925 07:30:18.478296 16027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:30:18.502739 16027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:30:18.505115 16027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:30:18.508263 16196 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.61:33901 every 8 connection(s)
I20260925 07:30:18.508246 16027 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.61:33901
I20260925 07:30:18.508710 16027 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/master-1/data/info.pb
I20260925 07:30:18.509452 16197 sys_catalog.cc:263] Verifying existing consensus state
I20260925 07:30:18.509915 16197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01d8c4dcbc1244b28cfa2b85723a0a54: Bootstrap starting.
I20260925 07:30:18.512303 16197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01d8c4dcbc1244b28cfa2b85723a0a54: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 07:30:18.512562 16197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01d8c4dcbc1244b28cfa2b85723a0a54: Bootstrap complete.
I20260925 07:30:18.512763 16197 raft_consensus.cc:348] T 00000000000000000000000000000000 P 01d8c4dcbc1244b28cfa2b85723a0a54 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "438b938b95aa4027a1d63ae68aef56ad" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 34429 } } peers { permanent_uuid: "01d8c4dcbc1244b28cfa2b85723a0a54" member_type: NON_VOTER last_known_addr { host: "127.9.66.61" port: 33901 } attrs { promote: true } }
I20260925 07:30:18.512859 16197 raft_consensus.cc:729] T 00000000000000000000000000000000 P 01d8c4dcbc1244b28cfa2b85723a0a54 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 01d8c4dcbc1244b28cfa2b85723a0a54, State: Initialized, Role: LEARNER
I20260925 07:30:18.512908 16197 consensus_queue.cc:260] T 00000000000000000000000000000000 P 01d8c4dcbc1244b28cfa2b85723a0a54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "438b938b95aa4027a1d63ae68aef56ad" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 34429 } } peers { permanent_uuid: "01d8c4dcbc1244b28cfa2b85723a0a54" member_type: NON_VOTER last_known_addr { host: "127.9.66.61" port: 33901 } attrs { promote: true } }
I20260925 07:30:18.513074 16197 sys_catalog.cc:564] T 00000000000000000000000000000000 P 01d8c4dcbc1244b28cfa2b85723a0a54 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:30:18.513053 16200 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01d8c4dcbc1244b28cfa2b85723a0a54 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "438b938b95aa4027a1d63ae68aef56ad" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 34429 } } peers { permanent_uuid: "01d8c4dcbc1244b28cfa2b85723a0a54" member_type: NON_VOTER last_known_addr { host: "127.9.66.61" port: 33901 } attrs { promote: true } } }
I20260925 07:30:18.513161 16200 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01d8c4dcbc1244b28cfa2b85723a0a54 [sys.catalog]: This master's current role is: LEARNER
I20260925 07:30:18.514499 16211 catalog_manager.cc:1259] Loaded cluster ID: 1f6a3af636aa4ddb853f1ed147c46d72
I20260925 07:30:18.514542 16211 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 01d8c4dcbc1244b28cfa2b85723a0a54: loading cluster ID for follower catalog manager: success
I20260925 07:30:18.515642 16211 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 01d8c4dcbc1244b28cfa2b85723a0a54: acquiring CA information for follower catalog manager: success
I20260925 07:30:18.515936 16211 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 01d8c4dcbc1244b28cfa2b85723a0a54: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 07:30:18.560024 16212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:18.553611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32964 (local address 127.9.66.61:33901)
0925 07:30:18.553746 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:18.553748 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:18.553755 (+ 7us) server_negotiation.cc:407] Connection header received
0925 07:30:18.553779 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:18.553780 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:18.553788 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:18.553867 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:30:18.554439 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:18.555089 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:18.555952 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:18.556139 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:18.556228 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:18.557476 (+ 1248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:30:18.557477 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:30:18.557477 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:30:18.557485 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:30:18.558794 (+ 1309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:18.559177 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:18.559178 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:18.559188 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:18.559247 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:18.559444 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:18.559444 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:18.559445 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:18.559568 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:30:18.559669 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:18.559816 (+ 147us) server_negotiation.cc:299] Negotiation successful
0925 07:30:18.559892 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":58,"threads_started":1}
WARNING: no policy specified for oryx/127.9.66.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.66.60@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.66.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/krb5kdc/oryx_127.9.66.60.keytab.
Entry for principal oryx/127.9.66.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/krb5kdc/oryx_127.9.66.60.keytab.
I20260925 07:30:18.585052 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.66.60:40627
--webserver_interface=127.9.66.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/krb5kdc/oryx_127.9.66.60.keytab
--principal=oryx/127.9.66.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:39737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.66.62:34429,127.9.66.61:33901,127.9.66.60:40627
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:30:18.681604 16215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:30:18.681807 16215 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:30:18.681828 16215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:30:18.682996 16215 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:30:18.683032 16215 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:30:18.683048 16215 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:30:18.683059 16215 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:30:18.683113 16215 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:30:18.684551 16215 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:39737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.9.66.62:34429,127.9.66.61:33901,127.9.66.60:40627
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/krb5kdc/oryx_127.9.66.60.keytab
--principal=oryx/127.9.66.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.66.60:40627
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.9.66.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16215
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:30:18.684799 16215 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:30:18.685029 16215 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:30:18.689819 16223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:30:18.689826 16220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:30:18.689829 16221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:30:18.690011 16215 server_base.cc:1034] running on GCE node
I20260925 07:30:18.690258 16215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:30:18.690522 16215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:30:18.691692 16215 hybrid_clock.cc:648] HybridClock initialized: now 1790321418691676 us; error 32 us; skew 500 ppm
Sep 25 07:30:18 dist-test-slave-57fn krb5kdc[15518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321418, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.66.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:30:18.694263 16215 init.cc:377] Logged in from keytab as oryx/127.9.66.60@KRBTEST.COM (short username oryx)
I20260925 07:30:18.694689 16215 webserver.cc:460] Webserver started at http://127.9.66.60:37209/ using document root <none> and password file <none>
I20260925 07:30:18.694867 16215 fs_manager.cc:359] Metadata directory not provided
I20260925 07:30:18.694905 16215 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:30:18.694985 16215 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:30:18.695875 16215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/master-2/data/instance:
uuid: "60014ba2af4c4663ab33a833de5b9113"
format_stamp: "Formatted at 2026-09-25 07:30:18 on dist-test-slave-57fn"
server_key: "cfde3430477009067d8b5e18906e38e9"
server_key_iv: "afb2ea5009494b419819be69e07e12ca"
server_key_version: "encryptionkey@0"
I20260925 07:30:18.696144 16215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/master-2/wal/instance:
uuid: "60014ba2af4c4663ab33a833de5b9113"
format_stamp: "Formatted at 2026-09-25 07:30:18 on dist-test-slave-57fn"
server_key: "cfde3430477009067d8b5e18906e38e9"
server_key_iv: "afb2ea5009494b419819be69e07e12ca"
server_key_version: "encryptionkey@0"
I20260925 07:30:18.697266 16215 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 07:30:18.697983 16230 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:18.698200 16215 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:18.698278 16215 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/master-2/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/master-2/wal
uuid: "60014ba2af4c4663ab33a833de5b9113"
format_stamp: "Formatted at 2026-09-25 07:30:18 on dist-test-slave-57fn"
server_key: "cfde3430477009067d8b5e18906e38e9"
server_key_iv: "afb2ea5009494b419819be69e07e12ca"
server_key_version: "encryptionkey@0"
I20260925 07:30:18.698325 16215 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:30:18.710959 16215 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:30:18.713003 16215 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:30:18.713117 16215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:30:18.717350 16215 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.60:40627
I20260925 07:30:18.717360 16282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.60:40627 every 8 connection(s)
I20260925 07:30:18.717753 16215 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/master-2/data/info.pb
I20260925 07:30:18.718779 16283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:30:18.719465 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 16215
I20260925 07:30:18.720777 16283 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.66.62" port: 34429 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 07:30:18 dist-test-slave-57fn krb5kdc[15518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321418, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.66.60@KRBTEST.COM for oryx/127.9.66.62@KRBTEST.COM
Sep 25 07:30:18 dist-test-slave-57fn krb5kdc[15518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321417, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.66.60@KRBTEST.COM
I20260925 07:30:18.729666 15621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:18.721574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33824 (local address 127.9.66.62:34429)
0925 07:30:18.721629 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:18.721631 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:18.721754 (+ 123us) server_negotiation.cc:407] Connection header received
0925 07:30:18.721794 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:18.721794 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:18.721803 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:18.721840 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:30:18.722417 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:18.722905 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:18.723851 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:18.724053 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:18.724114 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:18.726245 (+ 2131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:30:18.726246 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:30:18.726247 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:30:18.726254 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:30:18.728269 (+ 2015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:18.728658 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:18.728659 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:18.728659 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:18.728720 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:18.729000 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:18.729002 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:18.729002 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:18.729126 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:30:18.729215 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:18.729473 (+ 258us) server_negotiation.cc:299] Negotiation successful
0925 07:30:18.729580 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260925 07:30:18.729873 16286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:18.721382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34429 (local address 127.0.0.1:33824)
0925 07:30:18.721627 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:18.721663 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 07:30:18.721741 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:30:18.721878 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:30:18.721880 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:30:18.722224 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:30:18.722333 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:18.722338 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:18.723009 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:18.723010 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:18.723753 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:18.723754 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:18.724135 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:18.724136 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:18.724205 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:18.724760 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:30:18.724767 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:30:18.726157 (+ 1390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:30:18.728349 (+ 2192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:30:18.728354 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:30:18.728355 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:30:18.728581 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:30:18.728779 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:30:18.728780 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:30:18.728780 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:30:18.728927 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:30:18.729197 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:30:18.729203 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:30:18.729339 (+ 136us) client_negotiation.cc:769] Sending connection context
0925 07:30:18.729449 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 07:30:18.729609 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":118,"threads_started":1}
I20260925 07:30:18.730049 16285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:18.720075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43124 (local address 127.9.66.60:40627)
0925 07:30:18.720238 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:18.720245 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:18.720267 (+ 22us) server_negotiation.cc:407] Connection header received
0925 07:30:18.720743 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:18.720750 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:18.720796 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:18.720904 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:30:18.721745 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:18.722850 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:18.723580 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:18.723977 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:18.724050 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:18.726873 (+ 2823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:30:18.726879 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:30:18.726880 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:30:18.726886 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:30:18.728837 (+ 1951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:18.729174 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:18.729177 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:18.729177 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:18.729238 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:18.729441 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:18.729442 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:18.729442 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:18.729616 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:30:18.729785 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:18.729863 (+ 78us) server_negotiation.cc:299] Negotiation successful
0925 07:30:18.729931 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":73,"threads_started":1}
I20260925 07:30:18.730585 16283 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.66.61" port: 33901 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 07:30:18 dist-test-slave-57fn krb5kdc[15518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321418, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.66.60@KRBTEST.COM for oryx/127.9.66.61@KRBTEST.COM
I20260925 07:30:18.736618 16286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:18.730808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.61:33901 (local address 127.0.0.1:32976)
0925 07:30:18.730857 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:18.730866 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 07:30:18.730899 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:30:18.731090 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:30:18.731091 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:30:18.731221 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:30:18.731261 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:18.731262 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:18.731902 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:18.731903 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:18.732512 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:18.732514 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:18.732904 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:18.732905 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:18.732973 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:18.733276 (+ 303us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:30:18.733280 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:30:18.734372 (+ 1092us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:30:18.735662 (+ 1290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:30:18.735663 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:30:18.735663 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:30:18.735827 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:30:18.736028 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:30:18.736029 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:30:18.736029 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:30:18.736083 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:30:18.736334 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:30:18.736335 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:30:18.736373 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 07:30:18.736440 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 07:30:18.736528 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260925 07:30:18.736636 16212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:18.730907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32976 (local address 127.9.66.61:33901)
0925 07:30:18.730971 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:18.730972 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:18.730980 (+ 8us) server_negotiation.cc:407] Connection header received
0925 07:30:18.731004 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:18.731005 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:18.731016 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:18.731087 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:30:18.731346 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:18.731821 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:18.732596 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:18.732817 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:18.732903 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:18.734471 (+ 1568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:30:18.734473 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:30:18.734473 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:30:18.734479 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:30:18.735562 (+ 1083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:18.735902 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:18.735903 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:18.735903 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:18.735961 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:18.736156 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:18.736158 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:18.736158 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:18.736267 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:30:18.736344 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:18.736474 (+ 130us) server_negotiation.cc:299] Negotiation successful
0925 07:30:18.736551 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 07:30:18.737059 16283 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.66.60" port: 40627 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 07:30:18 dist-test-slave-57fn krb5kdc[15518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321418, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.66.60@KRBTEST.COM for oryx/127.9.66.60@KRBTEST.COM
I20260925 07:30:18.742817 16285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:18.737365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43132 (local address 127.9.66.60:40627)
0925 07:30:18.737398 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:18.737400 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:18.737407 (+ 7us) server_negotiation.cc:407] Connection header received
0925 07:30:18.737424 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:18.737424 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:18.737431 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:18.737474 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:30:18.737727 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:18.738145 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:18.738744 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:18.738966 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:18.739047 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:18.740490 (+ 1443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:30:18.740491 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:30:18.740491 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:30:18.740497 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:30:18.741731 (+ 1234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:18.742090 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:18.742091 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:18.742091 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:18.742144 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:18.742358 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:18.742359 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:18.742359 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:18.742458 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:30:18.742546 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:18.742640 (+ 94us) server_negotiation.cc:299] Negotiation successful
0925 07:30:18.742716 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 07:30:18.742905 16286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:18.737281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.60:40627 (local address 127.0.0.1:43132)
0925 07:30:18.737327 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:18.737336 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 07:30:18.737374 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:30:18.737475 (+ 101us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:30:18.737475 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:30:18.737621 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:30:18.737642 (+ 21us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:18.737643 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:18.738225 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:18.738225 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:18.738655 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:18.738656 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:18.739047 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:18.739048 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:18.739094 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:18.739391 (+ 297us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:30:18.739395 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:30:18.740406 (+ 1011us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:30:18.741851 (+ 1445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:30:18.741852 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:30:18.741853 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:30:18.742005 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:30:18.742207 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:30:18.742208 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:30:18.742209 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:30:18.742282 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:30:18.742525 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:30:18.742525 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:30:18.742559 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 07:30:18.742622 (+ 63us) client_negotiation.cc:240] Negotiation successful
0925 07:30:18.742782 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31,"mutex_wait_us":84}
I20260925 07:30:18.744122 16283 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60014ba2af4c4663ab33a833de5b9113: Bootstrap starting.
I20260925 07:30:18.745023 16283 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 60014ba2af4c4663ab33a833de5b9113: Neither blocks nor log segments found. Creating new log.
I20260925 07:30:18.745326 16283 log.cc:824] T 00000000000000000000000000000000 P 60014ba2af4c4663ab33a833de5b9113: Log is configured to *not* fsync() on all Append() calls
I20260925 07:30:18.746135 16283 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60014ba2af4c4663ab33a833de5b9113: No bootstrap required, opened a new log
I20260925 07:30:18.747599 16283 raft_consensus.cc:348] T 00000000000000000000000000000000 P 60014ba2af4c4663ab33a833de5b9113 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "438b938b95aa4027a1d63ae68aef56ad" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 34429 } } peers { permanent_uuid: "01d8c4dcbc1244b28cfa2b85723a0a54" member_type: VOTER last_known_addr { host: "127.9.66.61" port: 33901 } } peers { permanent_uuid: "60014ba2af4c4663ab33a833de5b9113" member_type: VOTER last_known_addr { host: "127.9.66.60" port: 40627 } }
I20260925 07:30:18.747714 16283 raft_consensus.cc:374] T 00000000000000000000000000000000 P 60014ba2af4c4663ab33a833de5b9113 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:30:18.747736 16283 raft_consensus.cc:729] T 00000000000000000000000000000000 P 60014ba2af4c4663ab33a833de5b9113 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60014ba2af4c4663ab33a833de5b9113, State: Initialized, Role: FOLLOWER
I20260925 07:30:18.747885 16283 consensus_queue.cc:260] T 00000000000000000000000000000000 P 60014ba2af4c4663ab33a833de5b9113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "438b938b95aa4027a1d63ae68aef56ad" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 34429 } } peers { permanent_uuid: "01d8c4dcbc1244b28cfa2b85723a0a54" member_type: VOTER last_known_addr { host: "127.9.66.61" port: 33901 } } peers { permanent_uuid: "60014ba2af4c4663ab33a833de5b9113" member_type: VOTER last_known_addr { host: "127.9.66.60" port: 40627 } }
I20260925 07:30:18.748214 16283 sys_catalog.cc:564] T 00000000000000000000000000000000 P 60014ba2af4c4663ab33a833de5b9113 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:30:18.748168 16288 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60014ba2af4c4663ab33a833de5b9113 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "438b938b95aa4027a1d63ae68aef56ad" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 34429 } } peers { permanent_uuid: "01d8c4dcbc1244b28cfa2b85723a0a54" member_type: VOTER last_known_addr { host: "127.9.66.61" port: 33901 } } peers { permanent_uuid: "60014ba2af4c4663ab33a833de5b9113" member_type: VOTER last_known_addr { host: "127.9.66.60" port: 40627 } } }
I20260925 07:30:18.748292 16288 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60014ba2af4c4663ab33a833de5b9113 [sys.catalog]: This master's current role is: FOLLOWER
W20260925 07:30:18.749817 16299 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 60014ba2af4c4663ab33a833de5b9113: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 07:30:18.749887 16299 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 07:30:18.752365 16215 master_runner.cc:428] Detected that this master 60014ba2af4c4663ab33a833de5b9113 is joining an existing cluster
I20260925 07:30:18.752414 16215 master_runner.cc:432] Initiating AddMaster RPC to add 127.9.66.60:40627
I20260925 07:30:18.760296 16305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:18.753647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.60:40627 (local address 127.0.0.1:43140)
0925 07:30:18.754046 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:18.754055 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 07:30:18.754119 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:30:18.754254 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:30:18.754254 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:30:18.754474 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:30:18.754537 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:18.754539 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:18.755353 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:18.755353 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:18.756215 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:18.756217 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:18.756656 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:18.756657 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:18.756731 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:18.757174 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:30:18.757179 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:30:18.757614 (+ 435us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:30:18.759273 (+ 1659us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:30:18.759274 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:30:18.759275 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:30:18.759441 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:30:18.759653 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:30:18.759653 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:30:18.759653 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:30:18.759715 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:30:18.759963 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:30:18.759964 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:30:18.760002 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 07:30:18.760070 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 07:30:18.760171 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":78,"threads_started":1}
I20260925 07:30:18.760339 16285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:18.753695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43140 (local address 127.9.66.60:40627)
0925 07:30:18.753728 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:18.753730 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:18.754119 (+ 389us) server_negotiation.cc:407] Connection header received
0925 07:30:18.754163 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:18.754164 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:18.754173 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:18.754221 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:30:18.754623 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:18.755273 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:18.756295 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:18.756548 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:18.756587 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:18.757706 (+ 1119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:30:18.757707 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:30:18.757707 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:30:18.757714 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:30:18.759146 (+ 1432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:18.759525 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:18.759526 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:18.759526 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:18.759579 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:18.759795 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:18.759796 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:18.759796 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:18.759895 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:30:18.759971 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:18.760099 (+ 128us) server_negotiation.cc:299] Negotiation successful
0925 07:30:18.760227 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":61,"server-negotiator.queue_time_us":18}
I20260925 07:30:18.777815 15621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:18.753499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33832 (local address 127.9.66.62:34429)
0925 07:30:18.753655 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:18.753658 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:18.753862 (+ 204us) server_negotiation.cc:407] Connection header received
0925 07:30:18.753917 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:18.753918 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:18.753934 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:18.753959 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:30:18.755429 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:18.756202 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:18.757467 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:18.757803 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:18.757880 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:18.774666 (+ 16786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:30:18.774668 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:30:18.774668 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:30:18.774676 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:30:18.776436 (+ 1760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:18.776943 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:18.776945 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:18.776945 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:18.777006 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:18.777182 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:18.777183 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:18.777183 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:18.777339 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:30:18.777405 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:18.777618 (+ 213us) server_negotiation.cc:299] Negotiation successful
0925 07:30:18.777700 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128}
I20260925 07:30:18.777812 16306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:18.753426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34429 (local address 127.0.0.1:33832)
0925 07:30:18.753780 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:18.753789 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 07:30:18.753851 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:30:18.754202 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:30:18.754202 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:30:18.755273 (+ 1071us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:30:18.755342 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:18.755344 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:18.756524 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:18.756525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:18.757371 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:18.757374 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:18.757891 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:18.757892 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:18.757970 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:18.774201 (+ 16231us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:30:18.774209 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:30:18.774541 (+ 332us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:30:18.776535 (+ 1994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:30:18.776536 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:30:18.776537 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:30:18.776855 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:30:18.777058 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:30:18.777059 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:30:18.777059 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:30:18.777119 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:30:18.777417 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:30:18.777418 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:30:18.777473 (+ 55us) client_negotiation.cc:769] Sending connection context
0925 07:30:18.777560 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 07:30:18.777681 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":101,"threads_started":1}
I20260925 07:30:18.777825 16212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:18.753598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32990 (local address 127.9.66.61:33901)
0925 07:30:18.753712 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:18.753714 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:18.753890 (+ 176us) server_negotiation.cc:407] Connection header received
0925 07:30:18.753913 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:18.753914 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:18.753924 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:18.753990 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:30:18.754390 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:18.755104 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:18.756249 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:18.756562 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:18.756604 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:18.774763 (+ 18159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:30:18.774764 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:30:18.774764 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:30:18.774773 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:30:18.776544 (+ 1771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:18.776992 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:18.776993 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:18.776993 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:18.777054 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:18.777223 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:18.777224 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:18.777225 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:18.777341 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:30:18.777450 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:18.777654 (+ 204us) server_negotiation.cc:299] Negotiation successful
0925 07:30:18.777743 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260925 07:30:18.778257 16307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:18.753427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.61:33901 (local address 127.0.0.1:32990)
0925 07:30:18.753791 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:18.753799 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 07:30:18.753851 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:30:18.754010 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:30:18.754011 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:30:18.754255 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:30:18.754328 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:18.754329 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:18.755195 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:18.755196 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:18.756151 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:18.756153 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:18.756678 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:18.756679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:18.756740 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:18.774147 (+ 17407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:30:18.774153 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:30:18.774656 (+ 503us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:30:18.776610 (+ 1954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:30:18.776611 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:30:18.776611 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:30:18.776872 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:30:18.777112 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:30:18.777112 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:30:18.777112 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:30:18.777166 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:30:18.777490 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:30:18.777491 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:30:18.777543 (+ 52us) client_negotiation.cc:769] Sending connection context
0925 07:30:18.778046 (+ 503us) client_negotiation.cc:240] Negotiation successful
0925 07:30:18.778147 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"mutex_wait_us":2,"thread_start_us":317,"threads_started":1}
I20260925 07:30:18.781247 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 15625
Sep 25 07:30:18 dist-test-slave-57fn krb5kdc[15518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321417, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.66.62@KRBTEST.COM for oryx/127.9.66.60@KRBTEST.COM
I20260925 07:30:18.786759 16285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:18.780160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43148 (local address 127.9.66.60:40627)
0925 07:30:18.780194 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:18.780196 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:18.780232 (+ 36us) server_negotiation.cc:407] Connection header received
0925 07:30:18.780257 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:18.780257 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:18.780265 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:18.780302 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:30:18.780612 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:18.781155 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:18.782066 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:18.782318 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:18.782399 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:18.784164 (+ 1765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:30:18.784165 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:30:18.784165 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:30:18.784173 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:30:18.785632 (+ 1459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:18.785994 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:18.785995 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:18.785995 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:18.786047 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:18.786258 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:18.786259 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:18.786260 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:18.786370 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:30:18.786442 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:18.786600 (+ 158us) server_negotiation.cc:299] Negotiation successful
0925 07:30:18.786670 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 07:30:18.786770 16121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:18.780097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.60:40627 (local address 127.0.0.1:43148)
0925 07:30:18.780145 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:18.780156 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 07:30:18.780191 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:30:18.780340 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:30:18.780341 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:30:18.780498 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:30:18.780543 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:18.780545 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:18.781241 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:18.781241 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:18.781977 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:18.781980 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:18.782398 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:18.782399 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:18.782477 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:18.782894 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:30:18.782899 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:30:18.784072 (+ 1173us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:30:18.785735 (+ 1663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:30:18.785736 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:30:18.785736 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:30:18.785907 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:30:18.786124 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:30:18.786125 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:30:18.786125 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:30:18.786183 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:30:18.786445 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:30:18.786446 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:30:18.786490 (+ 44us) client_negotiation.cc:769] Sending connection context
0925 07:30:18.786576 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 07:30:18.786672 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260925 07:30:18.788877 15563 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.9.66.60:40627: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "60014ba2af4c4663ab33a833de5b9113"
member_type: NON_VOTER
last_known_addr {
host: "127.9.66.60"
port: 40627
}
attrs {
promote: true
}
}
dest_uuid: "438b938b95aa4027a1d63ae68aef56ad"
cas_config_opid_index: 5
I20260925 07:30:18.789135 15563 consensus_queue.cc:237] T 00000000000000000000000000000000 P 438b938b95aa4027a1d63ae68aef56ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "438b938b95aa4027a1d63ae68aef56ad" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 34429 } } peers { permanent_uuid: "01d8c4dcbc1244b28cfa2b85723a0a54" member_type: NON_VOTER last_known_addr { host: "127.9.66.61" port: 33901 } attrs { promote: true } } peers { permanent_uuid: "60014ba2af4c4663ab33a833de5b9113" member_type: NON_VOTER last_known_addr { host: "127.9.66.60" port: 40627 } attrs { promote: true } }
I20260925 07:30:18.789211 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.66.1:36507
--local_ip_for_outbound_sockets=127.9.66.1
--tserver_master_addrs=127.9.66.62:34429
--webserver_port=44065
--webserver_interface=127.9.66.1
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/krb5kdc/oryx_127.9.66.1.keytab
--principal=oryx/127.9.66.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:39737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.9.66.62:34429,127.9.66.61:33901
--tserver_master_addrs=127.9.66.62:34429,127.9.66.61:33901,127.9.66.60:40627 with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 07:30:18.789954 16257 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 60014ba2af4c4663ab33a833de5b9113 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:30:18.790637 16122 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 438b938b95aa4027a1d63ae68aef56ad [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 60014ba2af4c4663ab33a833de5b9113 (127.9.66.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "438b938b95aa4027a1d63ae68aef56ad" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 34429 } } peers { permanent_uuid: "01d8c4dcbc1244b28cfa2b85723a0a54" member_type: NON_VOTER last_known_addr { host: "127.9.66.61" port: 33901 } attrs { promote: true } } peers { permanent_uuid: "60014ba2af4c4663ab33a833de5b9113" member_type: NON_VOTER last_known_addr { host: "127.9.66.60" port: 40627 } attrs { promote: true } } }
I20260925 07:30:18.790985 16257 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 60014ba2af4c4663ab33a833de5b9113 [term 1 FOLLOWER]: Refusing update from remote peer 438b938b95aa4027a1d63ae68aef56ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260925 07:30:18.791265 16123 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 438b938b95aa4027a1d63ae68aef56ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "60014ba2af4c4663ab33a833de5b9113" member_type: NON_VOTER last_known_addr { host: "127.9.66.60" port: 40627 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:30:18.791532 16122 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.9.66.60:40627
I20260925 07:30:18.791867 16123 sys_catalog.cc:455] T 00000000000000000000000000000000 P 438b938b95aa4027a1d63ae68aef56ad [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "438b938b95aa4027a1d63ae68aef56ad" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "438b938b95aa4027a1d63ae68aef56ad" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 34429 } } peers { permanent_uuid: "01d8c4dcbc1244b28cfa2b85723a0a54" member_type: NON_VOTER last_known_addr { host: "127.9.66.61" port: 33901 } attrs { promote: true } } peers { permanent_uuid: "60014ba2af4c4663ab33a833de5b9113" member_type: NON_VOTER last_known_addr { host: "127.9.66.60" port: 40627 } attrs { promote: true } } }
I20260925 07:30:18.791944 16123 sys_catalog.cc:458] T 00000000000000000000000000000000 P 438b938b95aa4027a1d63ae68aef56ad [sys.catalog]: This master's current role is: LEADER
I20260925 07:30:18.792083 16215 master.cc:537] Master@127.9.66.60:40627 shutting down...
I20260925 07:30:18.792694 16257 raft_consensus.cc:858] T 00000000000000000000000000000000 P 60014ba2af4c4663ab33a833de5b9113 [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "438b938b95aa4027a1d63ae68aef56ad" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 34429 } } peers { permanent_uuid: "01d8c4dcbc1244b28cfa2b85723a0a54" member_type: VOTER last_known_addr { host: "127.9.66.61" port: 33901 } } peers { permanent_uuid: "60014ba2af4c4663ab33a833de5b9113" member_type: VOTER last_known_addr { host: "127.9.66.60" port: 40627 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "438b938b95aa4027a1d63ae68aef56ad" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 34429 } } peers { permanent_uuid: "01d8c4dcbc1244b28cfa2b85723a0a54" member_type: NON_VOTER last_known_addr { host: "127.9.66.61" port: 33901 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "438b938b95aa4027a1d63ae68aef56ad" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 34429 } } peers { permanent_uuid: "01d8c4dcbc1244b28cfa2b85723a0a54" member_type: NON_VOTER last_known_addr { host: "127.9.66.61" port: 33901 } attrs { promote: true } } peers { permanent_uuid: "60014ba2af4c4663ab33a833de5b9113" member_type: NON_VOTER last_known_addr { host: "127.9.66.60" port: 40627 } attrs { promote: true } }
W20260925 07:30:18.792755 16257 raft_consensus.cc:1565] T 00000000000000000000000000000000 P 60014ba2af4c4663ab33a833de5b9113 [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "438b938b95aa4027a1d63ae68aef56ad" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 34429 } } peers { permanent_uuid: "01d8c4dcbc1244b28cfa2b85723a0a54" member_type: VOTER last_known_addr { host: "127.9.66.61" port: 33901 } } peers { permanent_uuid: "60014ba2af4c4663ab33a833de5b9113" member_type: VOTER last_known_addr { host: "127.9.66.60" port: 40627 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "438b938b95aa4027a1d63ae68aef56ad" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 34429 } } peers { permanent_uuid: "01d8c4dcbc1244b28cfa2b85723a0a54" member_type: NON_VOTER last_known_addr { host: "127.9.66.61" port: 33901 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "438b938b95aa4027a1d63ae68aef56ad" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 34429 } } peers { permanent_uuid: "01d8c4dcbc1244b28cfa2b85723a0a54" member_type: NON_VOTER last_known_addr { host: "127.9.66.61" port: 33901 } attrs { promote: true } } peers { permanent_uuid: "60014ba2af4c4663ab33a833de5b9113" member_type: NON_VOTER last_known_addr { host: "127.9.66.60" port: 40627 } attrs { promote: true } }
I20260925 07:30:18.793248 16257 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 60014ba2af4c4663ab33a833de5b9113 [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 01d8c4dcbc1244b28cfa2b85723a0a54 (127.9.66.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "438b938b95aa4027a1d63ae68aef56ad" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 34429 } } peers { permanent_uuid: "01d8c4dcbc1244b28cfa2b85723a0a54" member_type: NON_VOTER last_known_addr { host: "127.9.66.61" port: 33901 } attrs { promote: true } } }
I20260925 07:30:18.794181 16288 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60014ba2af4c4663ab33a833de5b9113 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 438b938b95aa4027a1d63ae68aef56ad. Latest consensus state: current_term: 1 leader_uuid: "438b938b95aa4027a1d63ae68aef56ad" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "438b938b95aa4027a1d63ae68aef56ad" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 34429 } } peers { permanent_uuid: "01d8c4dcbc1244b28cfa2b85723a0a54" member_type: NON_VOTER last_known_addr { host: "127.9.66.61" port: 33901 } attrs { promote: true } } }
I20260925 07:30:18.794265 16288 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60014ba2af4c4663ab33a833de5b9113 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260925 07:30:18.794294 16288 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60014ba2af4c4663ab33a833de5b9113 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "438b938b95aa4027a1d63ae68aef56ad" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "438b938b95aa4027a1d63ae68aef56ad" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 34429 } } peers { permanent_uuid: "01d8c4dcbc1244b28cfa2b85723a0a54" member_type: NON_VOTER last_known_addr { host: "127.9.66.61" port: 33901 } attrs { promote: true } } }
I20260925 07:30:18.794324 16288 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60014ba2af4c4663ab33a833de5b9113 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260925 07:30:18.794348 16288 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60014ba2af4c4663ab33a833de5b9113 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "438b938b95aa4027a1d63ae68aef56ad" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "438b938b95aa4027a1d63ae68aef56ad" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 34429 } } peers { permanent_uuid: "01d8c4dcbc1244b28cfa2b85723a0a54" member_type: NON_VOTER last_known_addr { host: "127.9.66.61" port: 33901 } attrs { promote: true } } }
I20260925 07:30:18.794378 16288 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60014ba2af4c4663ab33a833de5b9113 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260925 07:30:18.795941 16121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:18.791780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.61:33901 (local address 127.0.0.1:33006)
0925 07:30:18.791825 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:18.791836 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 07:30:18.791877 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:30:18.792288 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:30:18.792289 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:30:18.792290 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:30:18.792326 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:18.792328 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:18.793324 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:18.793324 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:18.794613 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:18.794616 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:18.795630 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:18.795631 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:18.795703 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:18.795734 (+ 31us) client_negotiation.cc:769] Sending connection context
0925 07:30:18.795788 (+ 54us) client_negotiation.cc:240] Negotiation successful
0925 07:30:18.795834 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260925 07:30:18.796358 16212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:18.791947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33006 (local address 127.9.66.61:33901)
0925 07:30:18.792073 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:18.792076 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:18.792084 (+ 8us) server_negotiation.cc:407] Connection header received
0925 07:30:18.792112 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:18.792112 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:18.792135 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:18.792195 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:30:18.792404 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:18.793105 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:18.794807 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:18.795445 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:18.795516 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:18.795594 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:18.796233 (+ 639us) server_negotiation.cc:299] Negotiation successful
0925 07:30:18.796276 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260925 07:30:18.796681 16170 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 01d8c4dcbc1244b28cfa2b85723a0a54 [term 1 LEARNER]: Refusing update from remote peer 438b938b95aa4027a1d63ae68aef56ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260925 07:30:18.796938 16123 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 438b938b95aa4027a1d63ae68aef56ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "01d8c4dcbc1244b28cfa2b85723a0a54" member_type: NON_VOTER last_known_addr { host: "127.9.66.61" port: 33901 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260925 07:30:18.797499 16170 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 01d8c4dcbc1244b28cfa2b85723a0a54 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 60014ba2af4c4663ab33a833de5b9113 (127.9.66.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "438b938b95aa4027a1d63ae68aef56ad" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 34429 } } peers { permanent_uuid: "01d8c4dcbc1244b28cfa2b85723a0a54" member_type: NON_VOTER last_known_addr { host: "127.9.66.61" port: 33901 } attrs { promote: true } } peers { permanent_uuid: "60014ba2af4c4663ab33a833de5b9113" member_type: NON_VOTER last_known_addr { host: "127.9.66.60" port: 40627 } attrs { promote: true } } }
I20260925 07:30:18.798576 16314 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01d8c4dcbc1244b28cfa2b85723a0a54 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "438b938b95aa4027a1d63ae68aef56ad" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "438b938b95aa4027a1d63ae68aef56ad" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 34429 } } peers { permanent_uuid: "01d8c4dcbc1244b28cfa2b85723a0a54" member_type: NON_VOTER last_known_addr { host: "127.9.66.61" port: 33901 } attrs { promote: true } } peers { permanent_uuid: "60014ba2af4c4663ab33a833de5b9113" member_type: NON_VOTER last_known_addr { host: "127.9.66.60" port: 40627 } attrs { promote: true } } }
I20260925 07:30:18.798727 16314 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01d8c4dcbc1244b28cfa2b85723a0a54 [sys.catalog]: This master's current role is: LEARNER
I20260925 07:30:18.799005 16215 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 60014ba2af4c4663ab33a833de5b9113 [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260925 07:30:18.799096 16215 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 60014ba2af4c4663ab33a833de5b9113 [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260925 07:30:18.799134 16215 tablet_replica.cc:330] T 00000000000000000000000000000000 P 60014ba2af4c4663ab33a833de5b9113: stopping tablet replica
I20260925 07:30:18.799162 16200 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01d8c4dcbc1244b28cfa2b85723a0a54 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 438b938b95aa4027a1d63ae68aef56ad. Latest consensus state: current_term: 1 leader_uuid: "438b938b95aa4027a1d63ae68aef56ad" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "438b938b95aa4027a1d63ae68aef56ad" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 34429 } } peers { permanent_uuid: "01d8c4dcbc1244b28cfa2b85723a0a54" member_type: NON_VOTER last_known_addr { host: "127.9.66.61" port: 33901 } attrs { promote: true } } peers { permanent_uuid: "60014ba2af4c4663ab33a833de5b9113" member_type: NON_VOTER last_known_addr { host: "127.9.66.60" port: 40627 } attrs { promote: true } } }
I20260925 07:30:18.799229 16200 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01d8c4dcbc1244b28cfa2b85723a0a54 [sys.catalog]: This master's current role is: LEARNER
I20260925 07:30:18.799357 16123 sys_catalog.cc:455] T 00000000000000000000000000000000 P 438b938b95aa4027a1d63ae68aef56ad [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "438b938b95aa4027a1d63ae68aef56ad" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "438b938b95aa4027a1d63ae68aef56ad" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 34429 } } peers { permanent_uuid: "01d8c4dcbc1244b28cfa2b85723a0a54" member_type: NON_VOTER last_known_addr { host: "127.9.66.61" port: 33901 } attrs { promote: true } } peers { permanent_uuid: "60014ba2af4c4663ab33a833de5b9113" member_type: NON_VOTER last_known_addr { host: "127.9.66.60" port: 40627 } attrs { promote: true } } }
I20260925 07:30:18.799458 16123 sys_catalog.cc:458] T 00000000000000000000000000000000 P 438b938b95aa4027a1d63ae68aef56ad [sys.catalog]: This master's current role is: LEADER
W20260925 07:30:18.799921 15548 consensus_peers.cc:487] T 00000000000000000000000000000000 P 438b938b95aa4027a1d63ae68aef56ad -> Peer 60014ba2af4c4663ab33a833de5b9113 (127.9.66.60:40627): Couldn't send request to peer 60014ba2af4c4663ab33a833de5b9113. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260925 07:30:18.805927 16122 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 438b938b95aa4027a1d63ae68aef56ad: attempting to promote NON_VOTER 01d8c4dcbc1244b28cfa2b85723a0a54 to VOTER
I20260925 07:30:18.806143 16122 consensus_queue.cc:237] T 00000000000000000000000000000000 P 438b938b95aa4027a1d63ae68aef56ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "438b938b95aa4027a1d63ae68aef56ad" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 34429 } } peers { permanent_uuid: "01d8c4dcbc1244b28cfa2b85723a0a54" member_type: VOTER last_known_addr { host: "127.9.66.61" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "60014ba2af4c4663ab33a833de5b9113" member_type: NON_VOTER last_known_addr { host: "127.9.66.60" port: 40627 } attrs { promote: true } }
I20260925 07:30:18.806661 16121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:18.806539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.60:40627 (local address 127.0.0.1:43158)
0925 07:30:18.806593 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:18.806631 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.60:40627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260925 07:30:18.806730 16170 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 01d8c4dcbc1244b28cfa2b85723a0a54 [term 1 LEARNER]: Refusing update from remote peer 438b938b95aa4027a1d63ae68aef56ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260925 07:30:18.806908 16318 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 438b938b95aa4027a1d63ae68aef56ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "01d8c4dcbc1244b28cfa2b85723a0a54" member_type: VOTER last_known_addr { host: "127.9.66.61" port: 33901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260925 07:30:18.807103 16121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:18.807029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.60:40627 (local address 127.0.0.1:43164)
0925 07:30:18.807062 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:18.807083 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.60:40627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260925 07:30:18.807278 15548 consensus_peers.cc:487] T 00000000000000000000000000000000 P 438b938b95aa4027a1d63ae68aef56ad -> Peer 60014ba2af4c4663ab33a833de5b9113 (127.9.66.60:40627): Couldn't send request to peer 60014ba2af4c4663ab33a833de5b9113. Status: Network error: Client connection negotiation failed: client connection to 127.9.66.60:40627: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260925 07:30:18.807649 16122 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 438b938b95aa4027a1d63ae68aef56ad [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 01d8c4dcbc1244b28cfa2b85723a0a54 (127.9.66.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "438b938b95aa4027a1d63ae68aef56ad" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 34429 } } peers { permanent_uuid: "01d8c4dcbc1244b28cfa2b85723a0a54" member_type: VOTER last_known_addr { host: "127.9.66.61" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "60014ba2af4c4663ab33a833de5b9113" member_type: NON_VOTER last_known_addr { host: "127.9.66.60" port: 40627 } attrs { promote: true } } }
I20260925 07:30:18.808364 16122 sys_catalog.cc:455] T 00000000000000000000000000000000 P 438b938b95aa4027a1d63ae68aef56ad [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "438b938b95aa4027a1d63ae68aef56ad" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "438b938b95aa4027a1d63ae68aef56ad" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 34429 } } peers { permanent_uuid: "01d8c4dcbc1244b28cfa2b85723a0a54" member_type: VOTER last_known_addr { host: "127.9.66.61" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "60014ba2af4c4663ab33a833de5b9113" member_type: NON_VOTER last_known_addr { host: "127.9.66.60" port: 40627 } attrs { promote: true } } }
I20260925 07:30:18.808439 16122 sys_catalog.cc:458] T 00000000000000000000000000000000 P 438b938b95aa4027a1d63ae68aef56ad [sys.catalog]: This master's current role is: LEADER
I20260925 07:30:18.808495 16122 sys_catalog.cc:455] T 00000000000000000000000000000000 P 438b938b95aa4027a1d63ae68aef56ad [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "438b938b95aa4027a1d63ae68aef56ad" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "438b938b95aa4027a1d63ae68aef56ad" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 34429 } } peers { permanent_uuid: "01d8c4dcbc1244b28cfa2b85723a0a54" member_type: VOTER last_known_addr { host: "127.9.66.61" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "60014ba2af4c4663ab33a833de5b9113" member_type: NON_VOTER last_known_addr { host: "127.9.66.60" port: 40627 } attrs { promote: true } } }
I20260925 07:30:18.808557 16122 sys_catalog.cc:458] T 00000000000000000000000000000000 P 438b938b95aa4027a1d63ae68aef56ad [sys.catalog]: This master's current role is: LEADER
I20260925 07:30:18.808786 16170 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 01d8c4dcbc1244b28cfa2b85723a0a54 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 01d8c4dcbc1244b28cfa2b85723a0a54 (127.9.66.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "438b938b95aa4027a1d63ae68aef56ad" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 34429 } } peers { permanent_uuid: "01d8c4dcbc1244b28cfa2b85723a0a54" member_type: VOTER last_known_addr { host: "127.9.66.61" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "60014ba2af4c4663ab33a833de5b9113" member_type: NON_VOTER last_known_addr { host: "127.9.66.60" port: 40627 } attrs { promote: true } } }
I20260925 07:30:18.809567 16200 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01d8c4dcbc1244b28cfa2b85723a0a54 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "438b938b95aa4027a1d63ae68aef56ad" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "438b938b95aa4027a1d63ae68aef56ad" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 34429 } } peers { permanent_uuid: "01d8c4dcbc1244b28cfa2b85723a0a54" member_type: VOTER last_known_addr { host: "127.9.66.61" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "60014ba2af4c4663ab33a833de5b9113" member_type: NON_VOTER last_known_addr { host: "127.9.66.60" port: 40627 } attrs { promote: true } } }
I20260925 07:30:18.809659 16200 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01d8c4dcbc1244b28cfa2b85723a0a54 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 07:30:18.814010 16215 master.cc:552] Master@127.9.66.60:40627 shutdown complete.
I20260925 07:30:18.814054 16215 master_runner.cc:305] Clearing existing system tablet
I20260925 07:30:18.814208 16215 fs_manager.cc:359] Metadata directory not provided
I20260925 07:30:18.814268 16215 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:30:18.814957 16215 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260925 07:30:18.815413 16320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:18.815574 16215 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:30:18.815649 16215 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:18.815678 16215 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/master-2/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/master-2/wal
uuid: "60014ba2af4c4663ab33a833de5b9113"
format_stamp: "Formatted at 2026-09-25 07:30:18 on dist-test-slave-57fn"
server_key: "cfde3430477009067d8b5e18906e38e9"
server_key_iv: "afb2ea5009494b419819be69e07e12ca"
server_key_version: "encryptionkey@0"
I20260925 07:30:18.816252 16215 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 60014ba2af4c4663ab33a833de5b9113: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 07:30:18.817473 16215 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 60014ba2af4c4663ab33a833de5b9113: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260925 07:30:18.817624 16215 log.cc:1196] T 00000000000000000000000000000000 P 60014ba2af4c4663ab33a833de5b9113: Deleting WAL directory at /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260925 07:30:18.817857 16215 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 60014ba2af4c4663ab33a833de5b9113: Deleting consensus metadata
I20260925 07:30:18.817996 16215 master_runner.cc:315] Copying system tablet from 127.9.66.62:34429
I20260925 07:30:18.818599 16215 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 60014ba2af4c4663ab33a833de5b9113: tablet copy: Beginning tablet copy session from remote peer at address 127.9.66.62:34429
I20260925 07:30:18.825133 15621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:18.818946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33842 (local address 127.9.66.62:34429)
0925 07:30:18.818993 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:18.818996 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:18.819118 (+ 122us) server_negotiation.cc:407] Connection header received
0925 07:30:18.819183 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:18.819183 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:18.819194 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:18.819250 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:30:18.819575 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:18.820061 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:18.820778 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:18.821056 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:18.821147 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:18.822488 (+ 1341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:30:18.822489 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:30:18.822490 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:30:18.822497 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:30:18.824023 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:18.824345 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:18.824346 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:18.824346 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:18.824415 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:18.824624 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:18.824625 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:18.824626 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:18.824752 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:30:18.824828 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:18.824979 (+ 151us) server_negotiation.cc:299] Negotiation successful
0925 07:30:18.825049 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 07:30:18.825142 16326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:18.818917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34429 (local address 127.0.0.1:33842)
0925 07:30:18.819035 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:18.819045 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 07:30:18.819131 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:30:18.819253 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:30:18.819253 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:30:18.819432 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:30:18.819485 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:18.819487 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:18.820146 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:18.820148 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:18.820681 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:18.820683 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:18.821148 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:18.821149 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:18.821216 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:18.822078 (+ 862us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:30:18.822082 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:30:18.822372 (+ 290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:30:18.824094 (+ 1722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:30:18.824095 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:30:18.824096 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:30:18.824268 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:30:18.824492 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:30:18.824493 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:30:18.824493 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:30:18.824550 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:30:18.824820 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:30:18.824821 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:30:18.824864 (+ 43us) client_negotiation.cc:769] Sending connection context
0925 07:30:18.824953 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 07:30:18.825050 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":61,"threads_started":1}
I20260925 07:30:18.825775 15583 tablet_copy_service.cc:140] P 438b938b95aa4027a1d63ae68aef56ad: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 60014ba2af4c4663ab33a833de5b9113 ({username='oryx', principal='oryx/127.9.66.60@KRBTEST.COM'} at 127.0.0.1:33842)
I20260925 07:30:18.825853 15583 tablet_copy_service.cc:161] P 438b938b95aa4027a1d63ae68aef56ad: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 60014ba2af4c4663ab33a833de5b9113 at {username='oryx', principal='oryx/127.9.66.60@KRBTEST.COM'} at 127.0.0.1:33842: session id = 60014ba2af4c4663ab33a833de5b9113-00000000000000000000000000000000
I20260925 07:30:18.826283 15583 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 438b938b95aa4027a1d63ae68aef56ad: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 07:30:18.826713 16215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260925 07:30:18.827769 16215 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 60014ba2af4c4663ab33a833de5b9113: tablet copy: Starting download of 0 data blocks...
I20260925 07:30:18.827924 16215 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 60014ba2af4c4663ab33a833de5b9113: tablet copy: Starting download of 1 WAL segments...
I20260925 07:30:18.828577 16215 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 60014ba2af4c4663ab33a833de5b9113: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 07:30:18.829267 15583 tablet_copy_service.cc:342] P 438b938b95aa4027a1d63ae68aef56ad: Request end of tablet copy session 60014ba2af4c4663ab33a833de5b9113-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.9.66.60@KRBTEST.COM'} at 127.0.0.1:33842
I20260925 07:30:18.829316 15583 tablet_copy_service.cc:433] P 438b938b95aa4027a1d63ae68aef56ad: ending tablet copy session 60014ba2af4c4663ab33a833de5b9113-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 60014ba2af4c4663ab33a833de5b9113
I20260925 07:30:18.831336 16215 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:30:18.832572 16327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:30:18.832587 16328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:30:18.834400 16330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:30:18.834472 16215 server_base.cc:1034] running on GCE node
I20260925 07:30:18.834664 16215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:30:18.834827 16215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:30:18.835950 16215 hybrid_clock.cc:648] HybridClock initialized: now 1790321418835943 us; error 25 us; skew 500 ppm
Sep 25 07:30:18 dist-test-slave-57fn krb5kdc[15518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321418, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.66.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:30:18.837306 16215 init.cc:377] Logged in from keytab as oryx/127.9.66.60@KRBTEST.COM (short username oryx)
I20260925 07:30:18.837567 16215 webserver.cc:460] Webserver started at http://127.9.66.60:44845/ using document root <none> and password file <none>
I20260925 07:30:18.837683 16215 fs_manager.cc:359] Metadata directory not provided
I20260925 07:30:18.837718 16215 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:30:18.838311 16215 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260925 07:30:18.838793 16337 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:18.838896 16215 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:18.838929 16215 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/master-2/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/master-2/wal
uuid: "60014ba2af4c4663ab33a833de5b9113"
format_stamp: "Formatted at 2026-09-25 07:30:18 on dist-test-slave-57fn"
server_key: "cfde3430477009067d8b5e18906e38e9"
server_key_iv: "afb2ea5009494b419819be69e07e12ca"
server_key_version: "encryptionkey@0"
I20260925 07:30:18.838985 16215 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:30:18.845019 16215 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:30:18.846724 16215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:30:18.849782 16215 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.60:40627
I20260925 07:30:18.850121 16215 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/master-2/data/info.pb
I20260925 07:30:18.853127 16389 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.60:40627 every 8 connection(s)
I20260925 07:30:18.853801 16390 sys_catalog.cc:263] Verifying existing consensus state
I20260925 07:30:18.854185 16390 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60014ba2af4c4663ab33a833de5b9113: Bootstrap starting.
I20260925 07:30:18.855921 16390 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60014ba2af4c4663ab33a833de5b9113: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 07:30:18.856133 16390 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60014ba2af4c4663ab33a833de5b9113: Bootstrap complete.
I20260925 07:30:18.856352 16390 raft_consensus.cc:348] T 00000000000000000000000000000000 P 60014ba2af4c4663ab33a833de5b9113 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "438b938b95aa4027a1d63ae68aef56ad" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 34429 } } peers { permanent_uuid: "01d8c4dcbc1244b28cfa2b85723a0a54" member_type: VOTER last_known_addr { host: "127.9.66.61" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "60014ba2af4c4663ab33a833de5b9113" member_type: NON_VOTER last_known_addr { host: "127.9.66.60" port: 40627 } attrs { promote: true } }
I20260925 07:30:18.856443 16390 raft_consensus.cc:729] T 00000000000000000000000000000000 P 60014ba2af4c4663ab33a833de5b9113 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 60014ba2af4c4663ab33a833de5b9113, State: Initialized, Role: LEARNER
I20260925 07:30:18.856488 16390 consensus_queue.cc:260] T 00000000000000000000000000000000 P 60014ba2af4c4663ab33a833de5b9113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "438b938b95aa4027a1d63ae68aef56ad" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 34429 } } peers { permanent_uuid: "01d8c4dcbc1244b28cfa2b85723a0a54" member_type: VOTER last_known_addr { host: "127.9.66.61" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "60014ba2af4c4663ab33a833de5b9113" member_type: NON_VOTER last_known_addr { host: "127.9.66.60" port: 40627 } attrs { promote: true } }
I20260925 07:30:18.856671 16390 sys_catalog.cc:564] T 00000000000000000000000000000000 P 60014ba2af4c4663ab33a833de5b9113 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:30:18.856667 16393 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60014ba2af4c4663ab33a833de5b9113 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "438b938b95aa4027a1d63ae68aef56ad" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 34429 } } peers { permanent_uuid: "01d8c4dcbc1244b28cfa2b85723a0a54" member_type: VOTER last_known_addr { host: "127.9.66.61" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "60014ba2af4c4663ab33a833de5b9113" member_type: NON_VOTER last_known_addr { host: "127.9.66.60" port: 40627 } attrs { promote: true } } }
I20260925 07:30:18.856899 16393 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60014ba2af4c4663ab33a833de5b9113 [sys.catalog]: This master's current role is: LEARNER
I20260925 07:30:18.858109 16404 catalog_manager.cc:1259] Loaded cluster ID: 1f6a3af636aa4ddb853f1ed147c46d72
I20260925 07:30:18.858803 16404 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 60014ba2af4c4663ab33a833de5b9113: loading cluster ID for follower catalog manager: success
I20260925 07:30:18.859503 16404 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 60014ba2af4c4663ab33a833de5b9113: acquiring CA information for follower catalog manager: success
I20260925 07:30:18.859769 16404 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 60014ba2af4c4663ab33a833de5b9113: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260925 07:30:18.906620 16308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:30:18.906812 16308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:30:18.906836 16308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:30:18.908087 16308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:30:18.908129 16308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:30:18.908156 16308 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.66.1
I20260925 07:30:18.909686 16308 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:39737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/krb5kdc/oryx_127.9.66.1.keytab
--principal=oryx/127.9.66.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.66.1:36507
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.66.1
--webserver_port=44065
--tserver_master_addrs=127.9.66.62:34429,127.9.66.61:33901,127.9.66.60:40627
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16308
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.66.1
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:30:18.909924 16308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:30:18.910135 16308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:30:18.915778 16412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:30:18.915853 16410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:30:18.915822 16409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:30:18.916399 16308 server_base.cc:1034] running on GCE node
I20260925 07:30:18.916610 16308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:30:18.916841 16308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:30:18.918009 16308 hybrid_clock.cc:648] HybridClock initialized: now 1790321418917995 us; error 26 us; skew 500 ppm
Sep 25 07:30:18 dist-test-slave-57fn krb5kdc[15518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321418, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.66.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:30:18.920884 16308 init.cc:377] Logged in from keytab as oryx/127.9.66.1@KRBTEST.COM (short username oryx)
I20260925 07:30:18.921370 16308 webserver.cc:460] Webserver started at http://127.9.66.1:44065/ using document root <none> and password file <none>
I20260925 07:30:18.921660 16308 fs_manager.cc:359] Metadata directory not provided
I20260925 07:30:18.921761 16308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:30:18.923106 16308 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:30:18.923722 16419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:18.923897 16308 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 07:30:18.923992 16308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/ts-0/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/ts-0/wal
uuid: "0d7f890138dd42a885a27044b0cb5f9d"
format_stamp: "Formatted at 2026-09-25 07:30:17 on dist-test-slave-57fn"
server_key: "8026a49b5cc4befa3eafd2147c36ebc6"
server_key_iv: "b6d308c8841e8c12ee1f0e4ef4575155"
server_key_version: "encryptionkey@0"
I20260925 07:30:18.924329 16308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:30:18.946436 16308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:30:18.948257 16308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:30:18.948418 16308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:30:18.948635 16308 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:30:18.948940 16308 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:30:18.948995 16308 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:18.949035 16308 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:30:18.949062 16308 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:18.954770 16308 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.1:36507
I20260925 07:30:18.954807 16532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.1:36507 every 8 connection(s)
I20260925 07:30:18.955156 16308 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/ts-0/data/info.pb
I20260925 07:30:18.956153 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 16308
I20260925 07:30:18.956308 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 15759
I20260925 07:30:18.962342 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.66.2:37737
--local_ip_for_outbound_sockets=127.9.66.2
--tserver_master_addrs=127.9.66.62:34429
--webserver_port=45989
--webserver_interface=127.9.66.2
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/krb5kdc/oryx_127.9.66.2.keytab
--principal=oryx/127.9.66.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:39737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.9.66.62:34429,127.9.66.61:33901
--tserver_master_addrs=127.9.66.62:34429,127.9.66.61:33901,127.9.66.60:40627 with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 07:30:18 dist-test-slave-57fn krb5kdc[15518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321418, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.66.1@KRBTEST.COM for oryx/127.9.66.60@KRBTEST.COM
Sep 25 07:30:18 dist-test-slave-57fn krb5kdc[15518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321418, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.66.1@KRBTEST.COM for oryx/127.9.66.62@KRBTEST.COM
Sep 25 07:30:18 dist-test-slave-57fn krb5kdc[15518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321418, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.66.1@KRBTEST.COM for oryx/127.9.66.61@KRBTEST.COM
I20260925 07:30:18.967409 16540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:18.956042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.1:56847 (local address 127.9.66.60:40627)
0925 07:30:18.956334 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:18.956336 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:18.956347 (+ 11us) server_negotiation.cc:407] Connection header received
0925 07:30:18.957416 (+ 1069us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:18.957417 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:18.957431 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:18.957490 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:30:18.958499 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:18.959660 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:18.960864 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:18.961187 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:18.961247 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:18.964071 (+ 2824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:30:18.964072 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:30:18.964072 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:30:18.964081 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:30:18.965783 (+ 1702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:18.966243 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:18.966244 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:18.966245 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:18.966314 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:18.966624 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:18.966625 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:18.966626 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:18.966775 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:30:18.966844 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:18.967166 (+ 322us) server_negotiation.cc:299] Negotiation successful
0925 07:30:18.967251 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":51,"threads_started":1}
I20260925 07:30:18.967881 16537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:18.955811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.60:40627 (local address 127.9.66.1:56847)
0925 07:30:18.955976 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:18.956016 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 07:30:18.957342 (+ 1326us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:30:18.957646 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:30:18.957650 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:30:18.958128 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:30:18.958396 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:18.958402 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:18.959753 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:18.959754 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:18.960770 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:18.960772 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:18.961299 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:18.961300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:18.961391 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:18.962289 (+ 898us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:30:18.962294 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:30:18.963968 (+ 1674us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:30:18.965871 (+ 1903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:30:18.965876 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:30:18.965879 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:30:18.966161 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:30:18.966391 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:30:18.966392 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:30:18.966392 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:30:18.966545 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:30:18.966849 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:30:18.966854 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:30:18.967013 (+ 159us) client_negotiation.cc:769] Sending connection context
0925 07:30:18.967140 (+ 127us) client_negotiation.cc:240] Negotiation successful
0925 07:30:18.967318 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"mutex_wait_us":21,"thread_start_us":95,"threads_started":1}
I20260925 07:30:18.968372 16533 heartbeater.cc:347] Connected to a master server at 127.9.66.60:40627
I20260925 07:30:18.968475 16533 heartbeater.cc:464] Registering TS with master...
I20260925 07:30:18.968674 15621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:18.956140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.1:39925 (local address 127.9.66.62:34429)
0925 07:30:18.956257 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:18.956258 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:18.956267 (+ 9us) server_negotiation.cc:407] Connection header received
0925 07:30:18.958032 (+ 1765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:18.958033 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:18.958045 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:18.958109 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:30:18.958538 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:18.959326 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:18.960848 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:18.961136 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:18.961189 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:18.964945 (+ 3756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:30:18.964946 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:30:18.964946 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:30:18.964952 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:30:18.966662 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:18.967769 (+ 1107us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:18.967770 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:18.967770 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:18.967822 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:18.968038 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:18.968039 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:18.968039 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:18.968156 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:30:18.968219 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:18.968543 (+ 324us) server_negotiation.cc:299] Negotiation successful
0925 07:30:18.968613 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260925 07:30:18.968691 16533 heartbeater.cc:511] Master 127.9.66.60:40627 requested a full tablet report, sending...
I20260925 07:30:18.969241 16354 ts_manager.cc:194] Registered new tserver with Master: 0d7f890138dd42a885a27044b0cb5f9d (127.9.66.1:36507)
I20260925 07:30:18.968988 16539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:18.955811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34429 (local address 127.9.66.1:39925)
0925 07:30:18.956050 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:18.956063 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 07:30:18.957220 (+ 1157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:30:18.958156 (+ 936us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:30:18.958157 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:30:18.958407 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:30:18.958475 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:18.958477 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:18.959431 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:18.959432 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:18.960753 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:18.960755 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:18.961248 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:18.961249 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:18.961387 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:18.962076 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:30:18.962086 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:30:18.964834 (+ 2748us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:30:18.967523 (+ 2689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:30:18.967524 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:30:18.967524 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:30:18.967693 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:30:18.967896 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:30:18.967897 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:30:18.967897 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:30:18.967958 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:30:18.968245 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:30:18.968246 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:30:18.968284 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 07:30:18.968346 (+ 62us) client_negotiation.cc:240] Negotiation successful
0925 07:30:18.968464 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"mutex_wait_us":41,"thread_start_us":90,"threads_started":1}
I20260925 07:30:18.969873 16534 heartbeater.cc:347] Connected to a master server at 127.9.66.62:34429
I20260925 07:30:18.969924 16534 heartbeater.cc:464] Registering TS with master...
I20260925 07:30:18.969986 16534 heartbeater.cc:511] Master 127.9.66.62:34429 requested a full tablet report, sending...
I20260925 07:30:18.970266 15563 ts_manager.cc:194] Re-registered known tserver with Master: 0d7f890138dd42a885a27044b0cb5f9d (127.9.66.1:36507)
I20260925 07:30:18.970629 15563 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.66.1@KRBTEST.COM'} at 127.9.66.1:39925
I20260925 07:30:18.974583 16212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:18.955871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.1:54047 (local address 127.9.66.61:33901)
0925 07:30:18.956192 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:18.956194 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:18.956842 (+ 648us) server_negotiation.cc:407] Connection header received
0925 07:30:18.957993 (+ 1151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:18.957994 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:18.958007 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:18.958038 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:30:18.958868 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:18.959558 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:18.960850 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:18.961137 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:18.961177 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:18.965268 (+ 4091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:30:18.965269 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:30:18.965270 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:30:18.965276 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:30:18.967428 (+ 2152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:18.973855 (+ 6427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:18.973856 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:18.973856 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:18.973910 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:18.974110 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:18.974111 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:18.974111 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:18.974229 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:30:18.974301 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:18.974432 (+ 131us) server_negotiation.cc:299] Negotiation successful
0925 07:30:18.974499 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294}
I20260925 07:30:18.974598 16538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:18.955811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.61:33901 (local address 127.9.66.1:54047)
0925 07:30:18.956769 (+ 958us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:18.956778 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 07:30:18.957769 (+ 991us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:30:18.958137 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:30:18.958138 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:30:18.958721 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:30:18.958777 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:18.958779 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:18.959639 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:18.959640 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:18.960753 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:18.960755 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:18.961260 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:18.961260 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:18.961387 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:18.962162 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:30:18.962167 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:30:18.965160 (+ 2993us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:30:18.973601 (+ 8441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:30:18.973602 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:30:18.973602 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:30:18.973775 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:30:18.973981 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:30:18.973982 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:30:18.973983 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:30:18.974044 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:30:18.974294 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:30:18.974295 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:30:18.974334 (+ 39us) client_negotiation.cc:769] Sending connection context
0925 07:30:18.974388 (+ 54us) client_negotiation.cc:240] Negotiation successful
0925 07:30:18.974496 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":920,"mutex_wait_us":11,"thread_start_us":87,"threads_started":1}
I20260925 07:30:18.974925 16535 heartbeater.cc:347] Connected to a master server at 127.9.66.61:33901
I20260925 07:30:18.974973 16535 heartbeater.cc:464] Registering TS with master...
I20260925 07:30:18.975034 16535 heartbeater.cc:511] Master 127.9.66.61:33901 requested a full tablet report, sending...
I20260925 07:30:18.975389 16159 ts_manager.cc:194] Registered new tserver with Master: 0d7f890138dd42a885a27044b0cb5f9d (127.9.66.1:36507)
W20260925 07:30:19.045660 16541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:30:19.045868 16541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:30:19.045904 16541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:30:19.047011 16541 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:30:19.047077 16541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:30:19.047122 16541 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.66.2
I20260925 07:30:19.048666 16541 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:39737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/krb5kdc/oryx_127.9.66.2.keytab
--principal=oryx/127.9.66.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.66.2:37737
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.66.2
--webserver_port=45989
--tserver_master_addrs=127.9.66.62:34429,127.9.66.61:33901,127.9.66.60:40627
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16541
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.66.2
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:30:19.048905 16541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:30:19.049119 16541 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:30:19.054003 16546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:30:19.054014 16549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:30:19.054014 16547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:30:19.054283 16541 server_base.cc:1034] running on GCE node
I20260925 07:30:19.054543 16541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:30:19.054797 16541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:30:19.055974 16541 hybrid_clock.cc:648] HybridClock initialized: now 1790321419055975 us; error 42 us; skew 500 ppm
Sep 25 07:30:19 dist-test-slave-57fn krb5kdc[15518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321419, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.66.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:30:19.058532 16541 init.cc:377] Logged in from keytab as oryx/127.9.66.2@KRBTEST.COM (short username oryx)
I20260925 07:30:19.058985 16541 webserver.cc:460] Webserver started at http://127.9.66.2:45989/ using document root <none> and password file <none>
I20260925 07:30:19.059218 16541 fs_manager.cc:359] Metadata directory not provided
I20260925 07:30:19.059301 16541 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:30:19.060550 16541 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:30:19.061118 16556 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:19.061324 16541 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 07:30:19.061419 16541 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/ts-1/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/ts-1/wal
uuid: "a90537ab87b444eda9751362d5b6febb"
format_stamp: "Formatted at 2026-09-25 07:30:18 on dist-test-slave-57fn"
server_key: "8b58671dc72370857e113d54d0e6b316"
server_key_iv: "cb79ab6a97ac7bd92b6a5c09125f5001"
server_key_version: "encryptionkey@0"
I20260925 07:30:19.061820 16541 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:30:19.090324 16541 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:30:19.092660 16541 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:30:19.092824 16541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:30:19.093050 16541 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:30:19.093356 16541 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:30:19.093411 16541 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:19.093459 16541 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:30:19.093485 16541 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:19.099537 16541 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.2:37737
I20260925 07:30:19.099576 16669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.2:37737 every 8 connection(s)
I20260925 07:30:19.099920 16541 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/ts-1/data/info.pb
Sep 25 07:30:19 dist-test-slave-57fn krb5kdc[15518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321419, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.66.2@KRBTEST.COM for oryx/127.9.66.60@KRBTEST.COM
Sep 25 07:30:19 dist-test-slave-57fn krb5kdc[15518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321419, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.66.2@KRBTEST.COM for oryx/127.9.66.61@KRBTEST.COM
Sep 25 07:30:19 dist-test-slave-57fn krb5kdc[15518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321419, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.66.2@KRBTEST.COM for oryx/127.9.66.62@KRBTEST.COM
I20260925 07:30:19.108875 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 16541
I20260925 07:30:19.109028 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 15893
I20260925 07:30:19.111208 16540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:19.100737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.2:52525 (local address 127.9.66.60:40627)
0925 07:30:19.100783 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:19.100785 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:19.101254 (+ 469us) server_negotiation.cc:407] Connection header received
0925 07:30:19.101655 (+ 401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:19.101656 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:19.101671 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:19.101729 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:30:19.102500 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:19.103252 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:19.104474 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:19.104775 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:19.104846 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:19.107430 (+ 2584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:30:19.107432 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:30:19.107432 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:30:19.107438 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:30:19.109474 (+ 2036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:19.110141 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:19.110143 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:19.110143 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:19.110194 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:19.110497 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:19.110497 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:19.110498 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:19.110622 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:30:19.110666 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:19.111029 (+ 363us) server_negotiation.cc:299] Negotiation successful
0925 07:30:19.111122 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 07:30:19.111202 16212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:19.101009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.2:51641 (local address 127.9.66.61:33901)
0925 07:30:19.101080 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:19.101082 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:19.101135 (+ 53us) server_negotiation.cc:407] Connection header received
0925 07:30:19.101709 (+ 574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:19.101710 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:19.101727 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:19.101795 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:30:19.102485 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:19.102973 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:19.104467 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:19.104775 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:19.104846 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:19.107932 (+ 3086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:30:19.107934 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:30:19.107934 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:30:19.107941 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:30:19.109582 (+ 1641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:19.110130 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:19.110131 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:19.110131 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:19.110190 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:19.110497 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:19.110497 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:19.110498 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:19.110634 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:30:19.110674 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:19.111029 (+ 355us) server_negotiation.cc:299] Negotiation successful
0925 07:30:19.111109 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 07:30:19.111693 16676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:19.100645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.60:40627 (local address 127.9.66.2:52525)
0925 07:30:19.101189 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:19.101197 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 07:30:19.101538 (+ 341us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:30:19.102202 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:30:19.102203 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:30:19.102348 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:30:19.102409 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:19.102410 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:19.103323 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:19.103324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:19.104356 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:19.104359 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:19.105206 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:19.105207 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:19.105276 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:19.105677 (+ 401us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:30:19.105682 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:30:19.107307 (+ 1625us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:30:19.109605 (+ 2298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:30:19.109611 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:30:19.109613 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:30:19.110039 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:30:19.110269 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:30:19.110270 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:30:19.110270 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:30:19.110416 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:30:19.110716 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:30:19.110722 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:30:19.110877 (+ 155us) client_negotiation.cc:769] Sending connection context
0925 07:30:19.111291 (+ 414us) client_negotiation.cc:240] Negotiation successful
0925 07:30:19.111478 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"mutex_wait_us":82,"thread_start_us":95,"threads_started":1}
I20260925 07:30:19.111714 16675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:19.100644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.61:33901 (local address 127.9.66.2:51641)
0925 07:30:19.101085 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:19.101094 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 07:30:19.101531 (+ 437us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:30:19.101836 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:30:19.101841 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:30:19.102176 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:30:19.102383 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:19.102388 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:19.103078 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:19.103079 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:19.104356 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:19.104359 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:19.104864 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:19.104865 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:19.104995 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:19.105486 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:30:19.105494 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:30:19.107842 (+ 2348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:30:19.109669 (+ 1827us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:30:19.109669 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:30:19.109670 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:30:19.110039 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:30:19.110269 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:30:19.110270 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:30:19.110270 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:30:19.110416 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:30:19.110717 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:30:19.110722 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:30:19.110877 (+ 155us) client_negotiation.cc:769] Sending connection context
0925 07:30:19.111291 (+ 414us) client_negotiation.cc:240] Negotiation successful
0925 07:30:19.111627 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":418,"mutex_wait_us":9,"thread_start_us":110,"threads_started":1}
I20260925 07:30:19.112123 16672 heartbeater.cc:347] Connected to a master server at 127.9.66.61:33901
I20260925 07:30:19.112222 16672 heartbeater.cc:464] Registering TS with master...
I20260925 07:30:19.112414 16672 heartbeater.cc:511] Master 127.9.66.61:33901 requested a full tablet report, sending...
I20260925 07:30:19.112881 16159 ts_manager.cc:194] Registered new tserver with Master: a90537ab87b444eda9751362d5b6febb (127.9.66.2:37737)
I20260925 07:30:19.113372 15621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:19.100945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.2:36547 (local address 127.9.66.62:34429)
0925 07:30:19.100996 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:19.100998 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:19.101008 (+ 10us) server_negotiation.cc:407] Connection header received
0925 07:30:19.101849 (+ 841us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:19.101851 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:19.101861 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:19.101917 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:30:19.102500 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:19.103246 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:19.105235 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:19.105540 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:19.105599 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:19.108478 (+ 2879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:30:19.108479 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:30:19.108480 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:30:19.108487 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:30:19.110464 (+ 1977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:19.110868 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:19.110869 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:19.110869 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:19.110921 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:19.111111 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:19.111112 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:19.111113 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:19.111242 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:30:19.111286 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:19.113108 (+ 1822us) server_negotiation.cc:299] Negotiation successful
0925 07:30:19.113214 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 07:30:19.113896 16674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:19.100647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34429 (local address 127.9.66.2:36547)
0925 07:30:19.100856 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:19.100897 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 07:30:19.101531 (+ 634us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:30:19.101943 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:30:19.101944 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:30:19.102197 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:30:19.102385 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:19.102392 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:19.104459 (+ 2067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:19.104459 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:19.105128 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:19.105130 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:19.105689 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:19.105690 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:19.105740 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:19.106132 (+ 392us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:30:19.106136 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:30:19.108375 (+ 2239us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:30:19.110552 (+ 2177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:30:19.110553 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:30:19.110554 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:30:19.110789 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:30:19.110990 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:30:19.110990 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:30:19.110991 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:30:19.111049 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:30:19.112938 (+ 1889us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:30:19.112939 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:30:19.112986 (+ 47us) client_negotiation.cc:769] Sending connection context
0925 07:30:19.113612 (+ 626us) client_negotiation.cc:240] Negotiation successful
0925 07:30:19.113727 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"mutex_wait_us":20,"thread_start_us":109,"threads_started":1}
I20260925 07:30:19.114300 16671 heartbeater.cc:347] Connected to a master server at 127.9.66.62:34429
I20260925 07:30:19.114351 16671 heartbeater.cc:464] Registering TS with master...
I20260925 07:30:19.114416 16671 heartbeater.cc:511] Master 127.9.66.62:34429 requested a full tablet report, sending...
I20260925 07:30:19.114645 15563 ts_manager.cc:194] Re-registered known tserver with Master: a90537ab87b444eda9751362d5b6febb (127.9.66.2:37737)
I20260925 07:30:19.115007 15563 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.66.2@KRBTEST.COM'} at 127.9.66.2:36547
I20260925 07:30:19.116321 16670 heartbeater.cc:347] Connected to a master server at 127.9.66.60:40627
I20260925 07:30:19.116364 16670 heartbeater.cc:464] Registering TS with master...
I20260925 07:30:19.116403 16670 heartbeater.cc:511] Master 127.9.66.60:40627 requested a full tablet report, sending...
I20260925 07:30:19.116681 16354 ts_manager.cc:194] Registered new tserver with Master: a90537ab87b444eda9751362d5b6febb (127.9.66.2:37737)
I20260925 07:30:19.116743 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.66.3:34477
--local_ip_for_outbound_sockets=127.9.66.3
--tserver_master_addrs=127.9.66.62:34429
--webserver_port=45871
--webserver_interface=127.9.66.3
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/krb5kdc/oryx_127.9.66.3.keytab
--principal=oryx/127.9.66.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:39737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.9.66.62:34429,127.9.66.61:33901
--tserver_master_addrs=127.9.66.62:34429,127.9.66.61:33901,127.9.66.60:40627 with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:30:19.196295 16677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:30:19.196523 16677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:30:19.196560 16677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:30:19.197831 16677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:30:19.197894 16677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:30:19.197938 16677 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.66.3
I20260925 07:30:19.199472 16677 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:39737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/krb5kdc/oryx_127.9.66.3.keytab
--principal=oryx/127.9.66.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.66.3:34477
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.66.3
--webserver_port=45871
--tserver_master_addrs=127.9.66.62:34429,127.9.66.61:33901,127.9.66.60:40627
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16677
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.66.3
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:30:19.199747 16677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:30:19.199982 16677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:30:19.205016 16682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:30:19.205015 16683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:30:19.205063 16685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:30:19.205297 16677 server_base.cc:1034] running on GCE node
I20260925 07:30:19.205482 16677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:30:19.205804 16677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:30:19.206954 16677 hybrid_clock.cc:648] HybridClock initialized: now 1790321419206939 us; error 20 us; skew 500 ppm
Sep 25 07:30:19 dist-test-slave-57fn krb5kdc[15518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321419, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.66.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:30:19.209369 16677 init.cc:377] Logged in from keytab as oryx/127.9.66.3@KRBTEST.COM (short username oryx)
I20260925 07:30:19.209811 16677 webserver.cc:460] Webserver started at http://127.9.66.3:45871/ using document root <none> and password file <none>
I20260925 07:30:19.209988 16677 fs_manager.cc:359] Metadata directory not provided
I20260925 07:30:19.210037 16677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:30:19.211340 16677 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 07:30:19.211880 16692 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:19.212072 16677 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 07:30:19.212139 16677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/ts-2/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/ts-2/wal
uuid: "9bfdbd8c5f3347a5a98a38d05d3fb756"
format_stamp: "Formatted at 2026-09-25 07:30:18 on dist-test-slave-57fn"
server_key: "7e55cb8e25bf6337372e1286fd7f7e52"
server_key_iv: "b583798a1b7468fbb2eda9481831d4c6"
server_key_version: "encryptionkey@0"
I20260925 07:30:19.212591 16677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:30:19.226800 16677 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:30:19.228693 16677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:30:19.228834 16677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:30:19.229063 16677 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:30:19.229382 16677 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:30:19.229419 16677 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:19.229439 16677 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:30:19.229488 16677 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:19.235147 16677 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.3:34477
I20260925 07:30:19.235221 16805 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.3:34477 every 8 connection(s)
I20260925 07:30:19.235560 16677 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/ts-2/data/info.pb
I20260925 07:30:19.241102 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 16677
Sep 25 07:30:19 dist-test-slave-57fn krb5kdc[15518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321419, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.66.3@KRBTEST.COM for oryx/127.9.66.61@KRBTEST.COM
Sep 25 07:30:19 dist-test-slave-57fn krb5kdc[15518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321419, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.66.3@KRBTEST.COM for oryx/127.9.66.60@KRBTEST.COM
Sep 25 07:30:19 dist-test-slave-57fn krb5kdc[15518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321419, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.66.3@KRBTEST.COM for oryx/127.9.66.62@KRBTEST.COM
I20260925 07:30:19.246819 16540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:19.236565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.3:59961 (local address 127.9.66.60:40627)
0925 07:30:19.236606 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:19.236608 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:19.236617 (+ 9us) server_negotiation.cc:407] Connection header received
0925 07:30:19.237240 (+ 623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:19.237241 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:19.237252 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:19.237301 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:30:19.238129 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:19.238815 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:19.239892 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:19.240179 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:19.240242 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:19.243369 (+ 3127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:30:19.243370 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:30:19.243371 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:30:19.243378 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:30:19.245110 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:19.245617 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:19.245618 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:19.245618 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:19.245686 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:19.245958 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:19.245958 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:19.245959 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:19.246089 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:30:19.246147 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:19.246640 (+ 493us) server_negotiation.cc:299] Negotiation successful
0925 07:30:19.246733 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 07:30:19.246999 16811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:19.236265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.60:40627 (local address 127.9.66.3:59961)
0925 07:30:19.236482 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:19.236523 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 07:30:19.237080 (+ 557us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:30:19.237349 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:30:19.237353 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:30:19.237786 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:30:19.238027 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:19.238033 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:19.238924 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:19.238924 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:19.239814 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:19.239817 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:19.240251 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:19.240251 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:19.240305 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:19.240824 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:30:19.240832 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:30:19.243265 (+ 2433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:30:19.245203 (+ 1938us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:30:19.245209 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:30:19.245211 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:30:19.245532 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:30:19.245760 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:30:19.245761 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:30:19.245762 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:30:19.245883 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:30:19.246172 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:30:19.246177 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:30:19.246341 (+ 164us) client_negotiation.cc:769] Sending connection context
0925 07:30:19.246438 (+ 97us) client_negotiation.cc:240] Negotiation successful
0925 07:30:19.246776 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"mutex_wait_us":40,"thread_start_us":80,"threads_started":1}
I20260925 07:30:19.247468 16806 heartbeater.cc:347] Connected to a master server at 127.9.66.60:40627
I20260925 07:30:19.247563 16806 heartbeater.cc:464] Registering TS with master...
I20260925 07:30:19.247736 16806 heartbeater.cc:511] Master 127.9.66.60:40627 requested a full tablet report, sending...
I20260925 07:30:19.248116 16812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:19.236265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.61:33901 (local address 127.9.66.3:52979)
0925 07:30:19.236683 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:19.236691 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 07:30:19.237093 (+ 402us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:30:19.237410 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:30:19.237410 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:30:19.237805 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:30:19.238026 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:19.238031 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:19.238729 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:19.238730 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:19.239654 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:19.239657 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:19.240138 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:19.240139 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:19.240268 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:19.240915 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:30:19.240939 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:30:19.244050 (+ 3111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:30:19.246433 (+ 2383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:30:19.246434 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:30:19.246434 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:30:19.246675 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:30:19.247518 (+ 843us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:30:19.247518 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:30:19.247519 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:30:19.247575 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:30:19.247817 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:30:19.247817 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:30:19.247853 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 07:30:19.247908 (+ 55us) client_negotiation.cc:240] Negotiation successful
0925 07:30:19.247993 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":398,"mutex_wait_us":43,"thread_start_us":69,"threads_started":1}
I20260925 07:30:19.248138 16354 ts_manager.cc:194] Registered new tserver with Master: 9bfdbd8c5f3347a5a98a38d05d3fb756 (127.9.66.3:34477)
I20260925 07:30:19.248446 16212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:19.236360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.3:52979 (local address 127.9.66.61:33901)
0925 07:30:19.236404 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:19.236407 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:19.236768 (+ 361us) server_negotiation.cc:407] Connection header received
0925 07:30:19.237308 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:19.237309 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:19.237320 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:19.237375 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:30:19.238117 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:19.238611 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:19.239768 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:19.240060 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:19.240123 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:19.244436 (+ 4313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:30:19.244437 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:30:19.244437 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:30:19.244444 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:30:19.246129 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:19.247384 (+ 1255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:19.247384 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:19.247385 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:19.247445 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:19.247636 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:19.247636 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:19.247637 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:19.247743 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:30:19.247794 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:19.248290 (+ 496us) server_negotiation.cc:299] Negotiation successful
0925 07:30:19.248359 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 07:30:19.249050 16808 heartbeater.cc:347] Connected to a master server at 127.9.66.61:33901
I20260925 07:30:19.249106 16808 heartbeater.cc:464] Registering TS with master...
I20260925 07:30:19.249161 16808 heartbeater.cc:511] Master 127.9.66.61:33901 requested a full tablet report, sending...
I20260925 07:30:19.249468 16159 ts_manager.cc:194] Registered new tserver with Master: 9bfdbd8c5f3347a5a98a38d05d3fb756 (127.9.66.3:34477)
I20260925 07:30:19.252061 15621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:19.236505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.3:47461 (local address 127.9.66.62:34429)
0925 07:30:19.236547 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:19.236549 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:19.236600 (+ 51us) server_negotiation.cc:407] Connection header received
0925 07:30:19.237192 (+ 592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:19.237194 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:19.237207 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:19.237257 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:30:19.238129 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:19.238825 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:19.240286 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:19.240487 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:19.240547 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:19.248997 (+ 8450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:30:19.248999 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:30:19.248999 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:30:19.249006 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:30:19.250893 (+ 1887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:19.251275 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:19.251276 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:19.251276 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:19.251331 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:19.251533 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:19.251534 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:19.251534 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:19.251651 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:30:19.251689 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:19.251898 (+ 209us) server_negotiation.cc:299] Negotiation successful
0925 07:30:19.251970 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 07:30:19.252424 16810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:19.236265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34429 (local address 127.9.66.3:47461)
0925 07:30:19.236541 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:19.236549 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 07:30:19.237080 (+ 531us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:30:19.237673 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:30:19.237674 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:30:19.237847 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:30:19.238027 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:19.238033 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:19.238924 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:19.238924 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:19.239817 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:19.239818 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:19.240555 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:19.240556 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:19.240601 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:19.242078 (+ 1477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:30:19.242083 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:30:19.248902 (+ 6819us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:30:19.250979 (+ 2077us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:30:19.250981 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:30:19.250981 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:30:19.251192 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:30:19.251402 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:30:19.251403 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:30:19.251403 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:30:19.251460 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:30:19.251735 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:30:19.251736 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:30:19.251790 (+ 54us) client_negotiation.cc:769] Sending connection context
0925 07:30:19.252187 (+ 397us) client_negotiation.cc:240] Negotiation successful
0925 07:30:19.252282 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"mutex_wait_us":51,"thread_start_us":76,"threads_started":1}
I20260925 07:30:19.252727 16807 heartbeater.cc:347] Connected to a master server at 127.9.66.62:34429
I20260925 07:30:19.252780 16807 heartbeater.cc:464] Registering TS with master...
I20260925 07:30:19.252852 16807 heartbeater.cc:511] Master 127.9.66.62:34429 requested a full tablet report, sending...
I20260925 07:30:19.253059 15563 ts_manager.cc:194] Re-registered known tserver with Master: 9bfdbd8c5f3347a5a98a38d05d3fb756 (127.9.66.3:34477)
I20260925 07:30:19.253564 15563 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.66.3@KRBTEST.COM'} at 127.9.66.3:47461
I20260925 07:30:19.253743 16817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:19.242134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33844 (local address 127.9.66.62:34429)
0925 07:30:19.242951 (+ 817us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:19.242952 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:19.242957 (+ 5us) server_negotiation.cc:407] Connection header received
0925 07:30:19.242973 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:19.242974 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:19.242983 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:19.243063 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:30:19.243836 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:19.244597 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:19.245610 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:19.245924 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:19.245962 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:19.248524 (+ 2562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:30:19.248526 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:30:19.248527 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:30:19.248534 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:30:19.251856 (+ 3322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:19.252396 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:19.252397 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:19.252397 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:19.252454 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:19.252915 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:19.252916 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:19.252917 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:19.253040 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:30:19.253164 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:19.253525 (+ 361us) server_negotiation.cc:299] Negotiation successful
0925 07:30:19.253596 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":796,"thread_start_us":72,"threads_started":1}
I20260925 07:30:19.253774 16821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:19.242524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43178 (local address 127.9.66.60:40627)
0925 07:30:19.243294 (+ 770us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:19.243296 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:19.243302 (+ 6us) server_negotiation.cc:407] Connection header received
0925 07:30:19.243322 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:19.243322 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:19.243333 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:19.243825 (+ 492us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:30:19.244185 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:19.244898 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:19.245935 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:19.246240 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:19.246301 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:19.250279 (+ 3978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:30:19.250281 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:30:19.250281 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:30:19.250288 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:30:19.251913 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:19.252479 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:19.252480 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:19.252480 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:19.252652 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:19.252851 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:19.252852 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:19.252853 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:19.252996 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:30:19.253469 (+ 473us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:19.253543 (+ 74us) server_negotiation.cc:299] Negotiation successful
0925 07:30:19.253624 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":750,"thread_start_us":55,"threads_started":1}
I20260925 07:30:19.253952 16819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:19.242279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33012 (local address 127.9.66.61:33901)
0925 07:30:19.242892 (+ 613us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:19.242894 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:19.243179 (+ 285us) server_negotiation.cc:407] Connection header received
0925 07:30:19.243425 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:19.243426 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:19.243440 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:19.243481 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:30:19.247267 (+ 3786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:19.248010 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:19.249006 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:19.249312 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:19.249379 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:19.250983 (+ 1604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:30:19.250984 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:30:19.250984 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:30:19.250991 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:30:19.252610 (+ 1619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:19.253095 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:19.253095 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:19.253096 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:19.253148 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:19.253347 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:19.253348 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:19.253348 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:19.253466 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:30:19.253564 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:19.253778 (+ 214us) server_negotiation.cc:299] Negotiation successful
0925 07:30:19.253854 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":596,"thread_start_us":62,"threads_started":1}
I20260925 07:30:19.255494 15563 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33844:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 07:30:19.257165 15563 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 07:30:19.264074 16121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:19.260664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.3:34477 (local address 127.0.0.1:53834)
0925 07:30:19.260742 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:19.260753 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 07:30:19.260794 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:30:19.261254 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:30:19.261255 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:30:19.261257 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:30:19.261293 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:19.261294 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:19.262167 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:19.262168 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:19.262939 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:19.262941 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:19.263831 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:19.263831 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:19.263889 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:19.263892 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 07:30:19.263950 (+ 58us) client_negotiation.cc:240] Negotiation successful
0925 07:30:19.264012 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260925 07:30:19.264098 16830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:19.260915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53834 (local address 127.9.66.3:34477)
0925 07:30:19.261055 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:19.261060 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:19.261068 (+ 8us) server_negotiation.cc:407] Connection header received
0925 07:30:19.261120 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:19.261122 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:19.261138 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:19.261360 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:30:19.261403 (+ 43us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:19.262079 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:19.263026 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:19.263721 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:19.263814 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:19.263873 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:19.263964 (+ 91us) server_negotiation.cc:299] Negotiation successful
0925 07:30:19.264030 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":66,"threads_started":1}
I20260925 07:30:19.265100 16827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:19.260783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.2:37737 (local address 127.0.0.1:37894)
0925 07:30:19.261035 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:19.261045 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 07:30:19.261091 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:30:19.261387 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:30:19.261388 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:30:19.261390 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:30:19.261423 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:19.261425 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:19.262755 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:19.262755 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:19.263950 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:19.263953 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:19.264842 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:19.264843 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:19.264905 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:19.264909 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 07:30:19.264965 (+ 56us) client_negotiation.cc:240] Negotiation successful
0925 07:30:19.265006 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"mutex_wait_us":37,"thread_start_us":56,"threads_started":1}
I20260925 07:30:19.265257 16831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:19.261045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37894 (local address 127.9.66.2:37737)
0925 07:30:19.261233 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:19.261237 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:19.261245 (+ 8us) server_negotiation.cc:407] Connection header received
0925 07:30:19.261279 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:19.261281 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:19.261295 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:19.261489 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:30:19.261574 (+ 85us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:19.262647 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:19.264048 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:19.264741 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:19.264829 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:19.264899 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:19.264978 (+ 79us) server_negotiation.cc:299] Negotiation successful
0925 07:30:19.265016 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":142,"threads_started":1}
I20260925 07:30:19.265269 16829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:19.260803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33734 (local address 127.9.66.1:36507)
0925 07:30:19.261071 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:19.261075 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:19.261085 (+ 10us) server_negotiation.cc:407] Connection header received
0925 07:30:19.261140 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:19.261142 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:19.261157 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:19.261246 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:30:19.261435 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:19.262774 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:19.263881 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:19.264703 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:19.264769 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:19.264840 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:19.265149 (+ 309us) server_negotiation.cc:299] Negotiation successful
0925 07:30:19.265191 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":76,"threads_started":1}
I20260925 07:30:19.265157 16740 tablet_service.cc:1467] Processing CreateTablet for tablet 23f00167abf74ae0812fe320d20081dc (DEFAULT_TABLE table=test-table [id=c42bea0452424a44a06d2ab8fa1a654d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:30:19.265306 16828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:19.260679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.1:36507 (local address 127.0.0.1:33734)
0925 07:30:19.260872 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:19.260882 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 07:30:19.260924 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:30:19.261313 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:30:19.261314 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:30:19.261316 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:30:19.261361 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:19.261363 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:19.262858 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:19.262859 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:19.263766 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:19.263768 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:19.264874 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:19.264875 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:19.264939 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:19.264944 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 07:30:19.264994 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 07:30:19.265044 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"mutex_wait_us":25,"thread_start_us":74,"threads_started":1}
I20260925 07:30:19.265596 16740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23f00167abf74ae0812fe320d20081dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:30:19.265877 16604 tablet_service.cc:1467] Processing CreateTablet for tablet 23f00167abf74ae0812fe320d20081dc (DEFAULT_TABLE table=test-table [id=c42bea0452424a44a06d2ab8fa1a654d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:30:19.265972 16467 tablet_service.cc:1467] Processing CreateTablet for tablet 23f00167abf74ae0812fe320d20081dc (DEFAULT_TABLE table=test-table [id=c42bea0452424a44a06d2ab8fa1a654d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:30:19.266196 16604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23f00167abf74ae0812fe320d20081dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:30:19.266235 16467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23f00167abf74ae0812fe320d20081dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:30:19.268280 16832 tablet_bootstrap.cc:492] T 23f00167abf74ae0812fe320d20081dc P 9bfdbd8c5f3347a5a98a38d05d3fb756: Bootstrap starting.
I20260925 07:30:19.268937 16832 tablet_bootstrap.cc:654] T 23f00167abf74ae0812fe320d20081dc P 9bfdbd8c5f3347a5a98a38d05d3fb756: Neither blocks nor log segments found. Creating new log.
I20260925 07:30:19.269220 16832 log.cc:824] T 23f00167abf74ae0812fe320d20081dc P 9bfdbd8c5f3347a5a98a38d05d3fb756: Log is configured to *not* fsync() on all Append() calls
I20260925 07:30:19.269481 16834 tablet_bootstrap.cc:492] T 23f00167abf74ae0812fe320d20081dc P 0d7f890138dd42a885a27044b0cb5f9d: Bootstrap starting.
I20260925 07:30:19.270084 16834 tablet_bootstrap.cc:654] T 23f00167abf74ae0812fe320d20081dc P 0d7f890138dd42a885a27044b0cb5f9d: Neither blocks nor log segments found. Creating new log.
I20260925 07:30:19.270197 16832 tablet_bootstrap.cc:492] T 23f00167abf74ae0812fe320d20081dc P 9bfdbd8c5f3347a5a98a38d05d3fb756: No bootstrap required, opened a new log
I20260925 07:30:19.270290 16832 ts_tablet_manager.cc:1397] T 23f00167abf74ae0812fe320d20081dc P 9bfdbd8c5f3347a5a98a38d05d3fb756: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 07:30:19.270442 16834 log.cc:824] T 23f00167abf74ae0812fe320d20081dc P 0d7f890138dd42a885a27044b0cb5f9d: Log is configured to *not* fsync() on all Append() calls
I20260925 07:30:19.270736 16833 tablet_bootstrap.cc:492] T 23f00167abf74ae0812fe320d20081dc P a90537ab87b444eda9751362d5b6febb: Bootstrap starting.
I20260925 07:30:19.271065 16834 tablet_bootstrap.cc:492] T 23f00167abf74ae0812fe320d20081dc P 0d7f890138dd42a885a27044b0cb5f9d: No bootstrap required, opened a new log
I20260925 07:30:19.271144 16834 ts_tablet_manager.cc:1397] T 23f00167abf74ae0812fe320d20081dc P 0d7f890138dd42a885a27044b0cb5f9d: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 07:30:19.271498 16833 tablet_bootstrap.cc:654] T 23f00167abf74ae0812fe320d20081dc P a90537ab87b444eda9751362d5b6febb: Neither blocks nor log segments found. Creating new log.
I20260925 07:30:19.271791 16833 log.cc:824] T 23f00167abf74ae0812fe320d20081dc P a90537ab87b444eda9751362d5b6febb: Log is configured to *not* fsync() on all Append() calls
I20260925 07:30:19.272324 16832 raft_consensus.cc:348] T 23f00167abf74ae0812fe320d20081dc P 9bfdbd8c5f3347a5a98a38d05d3fb756 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfdbd8c5f3347a5a98a38d05d3fb756" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 34477 } } peers { permanent_uuid: "0d7f890138dd42a885a27044b0cb5f9d" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 36507 } } peers { permanent_uuid: "a90537ab87b444eda9751362d5b6febb" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 37737 } }
I20260925 07:30:19.272496 16832 raft_consensus.cc:374] T 23f00167abf74ae0812fe320d20081dc P 9bfdbd8c5f3347a5a98a38d05d3fb756 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:30:19.272531 16833 tablet_bootstrap.cc:492] T 23f00167abf74ae0812fe320d20081dc P a90537ab87b444eda9751362d5b6febb: No bootstrap required, opened a new log
I20260925 07:30:19.272532 16832 raft_consensus.cc:729] T 23f00167abf74ae0812fe320d20081dc P 9bfdbd8c5f3347a5a98a38d05d3fb756 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bfdbd8c5f3347a5a98a38d05d3fb756, State: Initialized, Role: FOLLOWER
I20260925 07:30:19.272588 16833 ts_tablet_manager.cc:1397] T 23f00167abf74ae0812fe320d20081dc P a90537ab87b444eda9751362d5b6febb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 07:30:19.272666 16832 consensus_queue.cc:260] T 23f00167abf74ae0812fe320d20081dc P 9bfdbd8c5f3347a5a98a38d05d3fb756 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfdbd8c5f3347a5a98a38d05d3fb756" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 34477 } } peers { permanent_uuid: "0d7f890138dd42a885a27044b0cb5f9d" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 36507 } } peers { permanent_uuid: "a90537ab87b444eda9751362d5b6febb" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 37737 } }
I20260925 07:30:19.272920 16832 ts_tablet_manager.cc:1428] T 23f00167abf74ae0812fe320d20081dc P 9bfdbd8c5f3347a5a98a38d05d3fb756: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 07:30:19.273042 16807 heartbeater.cc:503] Master 127.9.66.62:34429 was elected leader, sending a full tablet report...
I20260925 07:30:19.273010 16834 raft_consensus.cc:348] T 23f00167abf74ae0812fe320d20081dc P 0d7f890138dd42a885a27044b0cb5f9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfdbd8c5f3347a5a98a38d05d3fb756" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 34477 } } peers { permanent_uuid: "0d7f890138dd42a885a27044b0cb5f9d" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 36507 } } peers { permanent_uuid: "a90537ab87b444eda9751362d5b6febb" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 37737 } }
I20260925 07:30:19.273370 16834 raft_consensus.cc:374] T 23f00167abf74ae0812fe320d20081dc P 0d7f890138dd42a885a27044b0cb5f9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:30:19.273411 16834 raft_consensus.cc:729] T 23f00167abf74ae0812fe320d20081dc P 0d7f890138dd42a885a27044b0cb5f9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d7f890138dd42a885a27044b0cb5f9d, State: Initialized, Role: FOLLOWER
I20260925 07:30:19.273558 16834 consensus_queue.cc:260] T 23f00167abf74ae0812fe320d20081dc P 0d7f890138dd42a885a27044b0cb5f9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfdbd8c5f3347a5a98a38d05d3fb756" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 34477 } } peers { permanent_uuid: "0d7f890138dd42a885a27044b0cb5f9d" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 36507 } } peers { permanent_uuid: "a90537ab87b444eda9751362d5b6febb" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 37737 } }
I20260925 07:30:19.273804 16834 ts_tablet_manager.cc:1428] T 23f00167abf74ae0812fe320d20081dc P 0d7f890138dd42a885a27044b0cb5f9d: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260925 07:30:19.273952 16534 heartbeater.cc:503] Master 127.9.66.62:34429 was elected leader, sending a full tablet report...
I20260925 07:30:19.274896 16833 raft_consensus.cc:348] T 23f00167abf74ae0812fe320d20081dc P a90537ab87b444eda9751362d5b6febb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfdbd8c5f3347a5a98a38d05d3fb756" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 34477 } } peers { permanent_uuid: "0d7f890138dd42a885a27044b0cb5f9d" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 36507 } } peers { permanent_uuid: "a90537ab87b444eda9751362d5b6febb" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 37737 } }
I20260925 07:30:19.275066 16833 raft_consensus.cc:374] T 23f00167abf74ae0812fe320d20081dc P a90537ab87b444eda9751362d5b6febb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:30:19.275112 16833 raft_consensus.cc:729] T 23f00167abf74ae0812fe320d20081dc P a90537ab87b444eda9751362d5b6febb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a90537ab87b444eda9751362d5b6febb, State: Initialized, Role: FOLLOWER
I20260925 07:30:19.275223 16833 consensus_queue.cc:260] T 23f00167abf74ae0812fe320d20081dc P a90537ab87b444eda9751362d5b6febb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfdbd8c5f3347a5a98a38d05d3fb756" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 34477 } } peers { permanent_uuid: "0d7f890138dd42a885a27044b0cb5f9d" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 36507 } } peers { permanent_uuid: "a90537ab87b444eda9751362d5b6febb" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 37737 } }
I20260925 07:30:19.275455 16833 ts_tablet_manager.cc:1428] T 23f00167abf74ae0812fe320d20081dc P a90537ab87b444eda9751362d5b6febb: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 07:30:19.275499 16671 heartbeater.cc:503] Master 127.9.66.62:34429 was elected leader, sending a full tablet report...
I20260925 07:30:19.281361 16839 raft_consensus.cc:482] T 23f00167abf74ae0812fe320d20081dc P 0d7f890138dd42a885a27044b0cb5f9d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 07:30:19.281567 16839 raft_consensus.cc:504] T 23f00167abf74ae0812fe320d20081dc P 0d7f890138dd42a885a27044b0cb5f9d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfdbd8c5f3347a5a98a38d05d3fb756" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 34477 } } peers { permanent_uuid: "0d7f890138dd42a885a27044b0cb5f9d" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 36507 } } peers { permanent_uuid: "a90537ab87b444eda9751362d5b6febb" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 37737 } }
I20260925 07:30:19.281889 16839 leader_election.cc:288] T 23f00167abf74ae0812fe320d20081dc P 0d7f890138dd42a885a27044b0cb5f9d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bfdbd8c5f3347a5a98a38d05d3fb756 (127.9.66.3:34477), a90537ab87b444eda9751362d5b6febb (127.9.66.2:37737)
I20260925 07:30:19.285398 16831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:19.282159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.1:56727 (local address 127.9.66.2:37737)
0925 07:30:19.282230 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:19.282232 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:19.282240 (+ 8us) server_negotiation.cc:407] Connection header received
0925 07:30:19.282317 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:19.282318 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:19.282327 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:19.282376 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:30:19.282483 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:19.283185 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:19.284547 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:19.285066 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:19.285137 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:19.285145 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:19.285306 (+ 161us) server_negotiation.cc:299] Negotiation successful
0925 07:30:19.285340 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 07:30:19.285398 16830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:19.282067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.1:37965 (local address 127.9.66.3:34477)
0925 07:30:19.282126 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:19.282128 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:19.282136 (+ 8us) server_negotiation.cc:407] Connection header received
0925 07:30:19.282194 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:19.282195 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:19.282206 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:19.282266 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:30:19.282421 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:19.283177 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:19.284547 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:19.285066 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:19.285137 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:19.285146 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:19.285306 (+ 160us) server_negotiation.cc:299] Negotiation successful
0925 07:30:19.285340 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 07:30:19.285416 16539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:19.282135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.2:37737 (local address 127.9.66.1:56727)
0925 07:30:19.282186 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:19.282195 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 07:30:19.282234 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:30:19.282376 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:30:19.282376 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:30:19.282378 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:30:19.282428 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:19.282429 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:19.283267 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:19.283267 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:19.284461 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:19.284464 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:19.285152 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:19.285153 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:19.285235 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:19.285238 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 07:30:19.285285 (+ 47us) client_negotiation.cc:240] Negotiation successful
0925 07:30:19.285329 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27,"mutex_wait_us":52}
I20260925 07:30:19.285429 16538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:19.282014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.3:34477 (local address 127.9.66.1:37965)
0925 07:30:19.282070 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:19.282085 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 07:30:19.282133 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:30:19.282284 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:30:19.282285 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:30:19.282287 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:30:19.282350 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:19.282352 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:19.283266 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:19.283267 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:19.284461 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:19.284464 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:19.285152 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:19.285153 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:19.285226 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:19.285230 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 07:30:19.285285 (+ 55us) client_negotiation.cc:240] Negotiation successful
0925 07:30:19.285329 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":62}
I20260925 07:30:19.285771 16760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f00167abf74ae0812fe320d20081dc" candidate_uuid: "0d7f890138dd42a885a27044b0cb5f9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfdbd8c5f3347a5a98a38d05d3fb756" is_pre_election: true
I20260925 07:30:19.285931 16760 raft_consensus.cc:2454] T 23f00167abf74ae0812fe320d20081dc P 9bfdbd8c5f3347a5a98a38d05d3fb756 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7f890138dd42a885a27044b0cb5f9d in term 0.
I20260925 07:30:19.285969 16624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f00167abf74ae0812fe320d20081dc" candidate_uuid: "0d7f890138dd42a885a27044b0cb5f9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a90537ab87b444eda9751362d5b6febb" is_pre_election: true
I20260925 07:30:19.286114 16624 raft_consensus.cc:2454] T 23f00167abf74ae0812fe320d20081dc P a90537ab87b444eda9751362d5b6febb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7f890138dd42a885a27044b0cb5f9d in term 0.
I20260925 07:30:19.286160 16420 leader_election.cc:302] T 23f00167abf74ae0812fe320d20081dc P 0d7f890138dd42a885a27044b0cb5f9d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7f890138dd42a885a27044b0cb5f9d, 9bfdbd8c5f3347a5a98a38d05d3fb756; no voters:
I20260925 07:30:19.286262 16839 raft_consensus.cc:2781] T 23f00167abf74ae0812fe320d20081dc P 0d7f890138dd42a885a27044b0cb5f9d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 07:30:19.286310 16839 raft_consensus.cc:482] T 23f00167abf74ae0812fe320d20081dc P 0d7f890138dd42a885a27044b0cb5f9d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 07:30:19.286339 16839 raft_consensus.cc:3037] T 23f00167abf74ae0812fe320d20081dc P 0d7f890138dd42a885a27044b0cb5f9d [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:30:19.286922 16839 raft_consensus.cc:504] T 23f00167abf74ae0812fe320d20081dc P 0d7f890138dd42a885a27044b0cb5f9d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfdbd8c5f3347a5a98a38d05d3fb756" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 34477 } } peers { permanent_uuid: "0d7f890138dd42a885a27044b0cb5f9d" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 36507 } } peers { permanent_uuid: "a90537ab87b444eda9751362d5b6febb" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 37737 } }
I20260925 07:30:19.287041 16839 leader_election.cc:288] T 23f00167abf74ae0812fe320d20081dc P 0d7f890138dd42a885a27044b0cb5f9d [CANDIDATE]: Term 1 election: Requested vote from peers 9bfdbd8c5f3347a5a98a38d05d3fb756 (127.9.66.3:34477), a90537ab87b444eda9751362d5b6febb (127.9.66.2:37737)
I20260925 07:30:19.287163 16760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f00167abf74ae0812fe320d20081dc" candidate_uuid: "0d7f890138dd42a885a27044b0cb5f9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfdbd8c5f3347a5a98a38d05d3fb756"
I20260925 07:30:19.287252 16760 raft_consensus.cc:3037] T 23f00167abf74ae0812fe320d20081dc P 9bfdbd8c5f3347a5a98a38d05d3fb756 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:30:19.287241 16624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f00167abf74ae0812fe320d20081dc" candidate_uuid: "0d7f890138dd42a885a27044b0cb5f9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a90537ab87b444eda9751362d5b6febb"
I20260925 07:30:19.287326 16624 raft_consensus.cc:3037] T 23f00167abf74ae0812fe320d20081dc P a90537ab87b444eda9751362d5b6febb [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:30:19.288056 16624 raft_consensus.cc:2454] T 23f00167abf74ae0812fe320d20081dc P a90537ab87b444eda9751362d5b6febb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7f890138dd42a885a27044b0cb5f9d in term 1.
I20260925 07:30:19.288209 16420 leader_election.cc:302] T 23f00167abf74ae0812fe320d20081dc P 0d7f890138dd42a885a27044b0cb5f9d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7f890138dd42a885a27044b0cb5f9d, a90537ab87b444eda9751362d5b6febb; no voters:
I20260925 07:30:19.288285 16839 raft_consensus.cc:2781] T 23f00167abf74ae0812fe320d20081dc P 0d7f890138dd42a885a27044b0cb5f9d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:30:19.288298 16760 raft_consensus.cc:2454] T 23f00167abf74ae0812fe320d20081dc P 9bfdbd8c5f3347a5a98a38d05d3fb756 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7f890138dd42a885a27044b0cb5f9d in term 1.
I20260925 07:30:19.288453 16839 raft_consensus.cc:686] T 23f00167abf74ae0812fe320d20081dc P 0d7f890138dd42a885a27044b0cb5f9d [term 1 LEADER]: Becoming Leader. State: Replica: 0d7f890138dd42a885a27044b0cb5f9d, State: Running, Role: LEADER
I20260925 07:30:19.288552 16839 consensus_queue.cc:237] T 23f00167abf74ae0812fe320d20081dc P 0d7f890138dd42a885a27044b0cb5f9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfdbd8c5f3347a5a98a38d05d3fb756" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 34477 } } peers { permanent_uuid: "0d7f890138dd42a885a27044b0cb5f9d" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 36507 } } peers { permanent_uuid: "a90537ab87b444eda9751362d5b6febb" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 37737 } }
I20260925 07:30:19.289273 15563 catalog_manager.cc:5495] T 23f00167abf74ae0812fe320d20081dc P 0d7f890138dd42a885a27044b0cb5f9d reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d7f890138dd42a885a27044b0cb5f9d (127.9.66.1). New cstate: current_term: 1 leader_uuid: "0d7f890138dd42a885a27044b0cb5f9d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bfdbd8c5f3347a5a98a38d05d3fb756" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 34477 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d7f890138dd42a885a27044b0cb5f9d" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 36507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a90537ab87b444eda9751362d5b6febb" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 37737 } health_report { overall_health: UNKNOWN } } }
I20260925 07:30:19.302011 16829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:19.299907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33746 (local address 127.9.66.1:36507)
0925 07:30:19.299945 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:19.299946 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:19.299953 (+ 7us) server_negotiation.cc:407] Connection header received
0925 07:30:19.300014 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:19.300015 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:19.300029 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:19.300084 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 07:30:19.300205 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:19.300643 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:19.301306 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:19.301473 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:19.301594 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:19.301732 (+ 138us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 07:30:19.301812 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 07:30:19.301865 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:19.301904 (+ 39us) server_negotiation.cc:299] Negotiation successful
0925 07:30:19.301919 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 07:30:19.303582 16760 raft_consensus.cc:1261] T 23f00167abf74ae0812fe320d20081dc P 9bfdbd8c5f3347a5a98a38d05d3fb756 [term 1 FOLLOWER]: Refusing update from remote peer 0d7f890138dd42a885a27044b0cb5f9d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:30:19.303599 16624 raft_consensus.cc:1261] T 23f00167abf74ae0812fe320d20081dc P a90537ab87b444eda9751362d5b6febb [term 1 FOLLOWER]: Refusing update from remote peer 0d7f890138dd42a885a27044b0cb5f9d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:30:19.303920 16841 consensus_queue.cc:1035] T 23f00167abf74ae0812fe320d20081dc P 0d7f890138dd42a885a27044b0cb5f9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bfdbd8c5f3347a5a98a38d05d3fb756" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 34477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:30:19.304018 16839 consensus_queue.cc:1035] T 23f00167abf74ae0812fe320d20081dc P 0d7f890138dd42a885a27044b0cb5f9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a90537ab87b444eda9751362d5b6febb" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 37737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:30:19.306732 16844 mvcc.cc:205] Tried to move back new op lower bound from 7333156533465022464 to 7333156533406584832. Current Snapshot: MvccSnapshot[applied={T|T < 7333156533465022464}]
I20260925 07:30:19.307197 16845 mvcc.cc:205] Tried to move back new op lower bound from 7333156533465022464 to 7333156533406584832. Current Snapshot: MvccSnapshot[applied={T|T < 7333156533465022464}]
I20260925 07:30:19.310180 15563 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33844:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 07:30:19.310278 15563 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33844:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 07:30:19.312366 15563 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 438b938b95aa4027a1d63ae68aef56ad: Sending DeleteTablet for 3 replicas of tablet 23f00167abf74ae0812fe320d20081dc
I20260925 07:30:19.312634 16467 tablet_service.cc:1514] Processing DeleteTablet for tablet 23f00167abf74ae0812fe320d20081dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:30:19 UTC) from {username='oryx', principal='oryx/127.9.66.62@KRBTEST.COM'} at 127.0.0.1:33734
I20260925 07:30:19.312721 16604 tablet_service.cc:1514] Processing DeleteTablet for tablet 23f00167abf74ae0812fe320d20081dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:30:19 UTC) from {username='oryx', principal='oryx/127.9.66.62@KRBTEST.COM'} at 127.0.0.1:37894
I20260925 07:30:19.312860 16854 tablet_replica.cc:330] T 23f00167abf74ae0812fe320d20081dc P 0d7f890138dd42a885a27044b0cb5f9d: stopping tablet replica
I20260925 07:30:19.312892 16740 tablet_service.cc:1514] Processing DeleteTablet for tablet 23f00167abf74ae0812fe320d20081dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:30:19 UTC) from {username='oryx', principal='oryx/127.9.66.62@KRBTEST.COM'} at 127.0.0.1:53834
I20260925 07:30:19.312997 16854 raft_consensus.cc:2229] T 23f00167abf74ae0812fe320d20081dc P 0d7f890138dd42a885a27044b0cb5f9d [term 1 LEADER]: Raft consensus shutting down.
I20260925 07:30:19.313107 16854 raft_consensus.cc:2258] T 23f00167abf74ae0812fe320d20081dc P 0d7f890138dd42a885a27044b0cb5f9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:30:19.313366 16856 tablet_replica.cc:330] T 23f00167abf74ae0812fe320d20081dc P 9bfdbd8c5f3347a5a98a38d05d3fb756: stopping tablet replica
I20260925 07:30:19.313566 16856 raft_consensus.cc:2229] T 23f00167abf74ae0812fe320d20081dc P 9bfdbd8c5f3347a5a98a38d05d3fb756 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 07:30:19.313617 16854 ts_tablet_manager.cc:1905] T 23f00167abf74ae0812fe320d20081dc P 0d7f890138dd42a885a27044b0cb5f9d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 07:30:19.313787 16856 raft_consensus.cc:2258] T 23f00167abf74ae0812fe320d20081dc P 9bfdbd8c5f3347a5a98a38d05d3fb756 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:30:19.314105 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 15531
I20260925 07:30:19.314332 16856 ts_tablet_manager.cc:1905] T 23f00167abf74ae0812fe320d20081dc P 9bfdbd8c5f3347a5a98a38d05d3fb756: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 07:30:19.316327 16856 ts_tablet_manager.cc:1918] T 23f00167abf74ae0812fe320d20081dc P 9bfdbd8c5f3347a5a98a38d05d3fb756: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 07:30:19.316396 16856 log.cc:1196] T 23f00167abf74ae0812fe320d20081dc P 9bfdbd8c5f3347a5a98a38d05d3fb756: Deleting WAL directory at /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/ts-2/wal/wals/23f00167abf74ae0812fe320d20081dc
I20260925 07:30:19.316674 16856 ts_tablet_manager.cc:1939] T 23f00167abf74ae0812fe320d20081dc P 9bfdbd8c5f3347a5a98a38d05d3fb756: Deleting consensus metadata
I20260925 07:30:19.317019 16854 ts_tablet_manager.cc:1918] T 23f00167abf74ae0812fe320d20081dc P 0d7f890138dd42a885a27044b0cb5f9d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 07:30:19.317075 16854 log.cc:1196] T 23f00167abf74ae0812fe320d20081dc P 0d7f890138dd42a885a27044b0cb5f9d: Deleting WAL directory at /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/ts-0/wal/wals/23f00167abf74ae0812fe320d20081dc
I20260925 07:30:19.317328 16854 ts_tablet_manager.cc:1939] T 23f00167abf74ae0812fe320d20081dc P 0d7f890138dd42a885a27044b0cb5f9d: Deleting consensus metadata
I20260925 07:30:19.321653 16855 tablet_replica.cc:330] T 23f00167abf74ae0812fe320d20081dc P a90537ab87b444eda9751362d5b6febb: stopping tablet replica
I20260925 07:30:19.321857 16855 raft_consensus.cc:2229] T 23f00167abf74ae0812fe320d20081dc P a90537ab87b444eda9751362d5b6febb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 07:30:19.321973 16855 raft_consensus.cc:2258] T 23f00167abf74ae0812fe320d20081dc P a90537ab87b444eda9751362d5b6febb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:30:19.322585 16855 ts_tablet_manager.cc:1905] T 23f00167abf74ae0812fe320d20081dc P a90537ab87b444eda9751362d5b6febb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 07:30:19.324156 16855 ts_tablet_manager.cc:1918] T 23f00167abf74ae0812fe320d20081dc P a90537ab87b444eda9751362d5b6febb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 07:30:19.324249 16855 log.cc:1196] T 23f00167abf74ae0812fe320d20081dc P a90537ab87b444eda9751362d5b6febb: Deleting WAL directory at /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0/minicluster-data/ts-1/wal/wals/23f00167abf74ae0812fe320d20081dc
W20260925 07:30:19.324311 16694 connection.cc:535] server connection from 127.0.0.1:53834 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 07:30:19.324528 16855 ts_tablet_manager.cc:1939] T 23f00167abf74ae0812fe320d20081dc P a90537ab87b444eda9751362d5b6febb: Deleting consensus metadata
W20260925 07:30:19.324637 16422 connection.cc:535] server connection from 127.0.0.1:33734 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260925 07:30:19.324928 16557 connection.cc:441] server connection from 127.0.0.1:37894 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37894 (request call id 1) could send its response
I20260925 07:30:19.327775 16821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:19.313664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43186 (local address 127.9.66.60:40627)
0925 07:30:19.313735 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:19.313737 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:19.316341 (+ 2604us) server_negotiation.cc:407] Connection header received
0925 07:30:19.316434 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:19.316435 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:19.316447 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:19.316475 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:30:19.319702 (+ 3227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:19.320497 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:19.321728 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:19.322103 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:19.322148 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:19.324171 (+ 2023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:30:19.324172 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:30:19.324173 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:30:19.324180 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:30:19.326327 (+ 2147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:19.326715 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:19.326716 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:19.326716 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:19.326775 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:19.327103 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:19.327104 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:19.327104 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:19.327252 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:30:19.327389 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:19.327597 (+ 208us) server_negotiation.cc:299] Negotiation successful
0925 07:30:19.327679 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 07:30:19.334355 16540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:19.326528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43198 (local address 127.9.66.60:40627)
0925 07:30:19.326566 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:19.326568 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:19.326575 (+ 7us) server_negotiation.cc:407] Connection header received
0925 07:30:19.326594 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:19.326594 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:19.326603 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:19.326659 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:30:19.327624 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:19.328289 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:19.329341 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:19.329659 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:19.329736 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:19.331382 (+ 1646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:30:19.331384 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:30:19.331384 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:30:19.331391 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:30:19.333050 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:19.333438 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:19.333439 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:19.333440 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:19.333498 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:19.333753 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:19.333754 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:19.333754 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:19.333865 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:30:19.333964 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:19.334176 (+ 212us) server_negotiation.cc:299] Negotiation successful
0925 07:30:19.334257 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 07:30:19.334447 16819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:19.326897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33022 (local address 127.9.66.61:33901)
0925 07:30:19.326938 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:19.326941 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:19.326948 (+ 7us) server_negotiation.cc:407] Connection header received
0925 07:30:19.326971 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:19.326972 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:19.326986 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:19.327049 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:30:19.327708 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:19.328481 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:19.329725 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:19.330034 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:19.330126 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:19.331408 (+ 1282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:30:19.331409 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:30:19.331409 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:30:19.331416 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:30:19.333222 (+ 1806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:19.333639 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:19.333640 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:19.333641 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:19.333694 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:19.333909 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:19.333910 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:19.333910 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:19.334037 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:30:19.334132 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:19.334277 (+ 145us) server_negotiation.cc:299] Negotiation successful
0925 07:30:19.334352 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 07:30:20.308398 16864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:20.308066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34429 (local address 127.9.66.2:56355)
0925 07:30:20.308261 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:20.308341 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":61,"threads_started":1}
W20260925 07:30:20.308612 16671 heartbeater.cc:644] Failed to heartbeat to 127.9.66.62:34429 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.9.66.62:34429: connect: Connection refused (error 111)
I20260925 07:30:20.308615 16865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:20.308262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34429 (local address 127.9.66.3:34803)
0925 07:30:20.308420 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:20.308486 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":48,"threads_started":1}
I20260925 07:30:20.308820 16866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:20.308301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34429 (local address 127.9.66.1:54437)
0925 07:30:20.308439 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:20.308643 (+ 204us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":55,"threads_started":1}
W20260925 07:30:20.308822 16807 heartbeater.cc:644] Failed to heartbeat to 127.9.66.62:34429 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.9.66.62:34429: connect: Connection refused (error 111)
W20260925 07:30:20.309099 16534 heartbeater.cc:644] Failed to heartbeat to 127.9.66.62:34429 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.9.66.62:34429: connect: Connection refused (error 111)
I20260925 07:30:20.812692 16867 raft_consensus.cc:482] T 00000000000000000000000000000000 P 01d8c4dcbc1244b28cfa2b85723a0a54 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 438b938b95aa4027a1d63ae68aef56ad)
I20260925 07:30:20.812850 16867 raft_consensus.cc:504] T 00000000000000000000000000000000 P 01d8c4dcbc1244b28cfa2b85723a0a54 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "438b938b95aa4027a1d63ae68aef56ad" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 34429 } } peers { permanent_uuid: "01d8c4dcbc1244b28cfa2b85723a0a54" member_type: VOTER last_known_addr { host: "127.9.66.61" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "60014ba2af4c4663ab33a833de5b9113" member_type: NON_VOTER last_known_addr { host: "127.9.66.60" port: 40627 } attrs { promote: true } }
I20260925 07:30:20.813102 16867 leader_election.cc:288] T 00000000000000000000000000000000 P 01d8c4dcbc1244b28cfa2b85723a0a54 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 438b938b95aa4027a1d63ae68aef56ad (127.9.66.62:34429)
I20260925 07:30:20.813409 16868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:20.813230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34429 (local address 127.0.0.1:34250)
0925 07:30:20.813309 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:20.813354 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64,"thread_start_us":42,"threads_started":1}
W20260925 07:30:20.813549 16147 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.66.62:34429: connect: Connection refused (error 111)
I20260925 07:30:20.813942 16868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:20.813847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34429 (local address 127.0.0.1:34254)
0925 07:30:20.813894 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:20.813923 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260925 07:30:20.814059 16147 leader_election.cc:334] T 00000000000000000000000000000000 P 01d8c4dcbc1244b28cfa2b85723a0a54 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 438b938b95aa4027a1d63ae68aef56ad (127.9.66.62:34429): Network error: Client connection negotiation failed: client connection to 127.9.66.62:34429: connect: Connection refused (error 111)
I20260925 07:30:20.814113 16147 leader_election.cc:302] T 00000000000000000000000000000000 P 01d8c4dcbc1244b28cfa2b85723a0a54 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 01d8c4dcbc1244b28cfa2b85723a0a54; no voters: 438b938b95aa4027a1d63ae68aef56ad
I20260925 07:30:20.814229 16867 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 01d8c4dcbc1244b28cfa2b85723a0a54 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 07:30:21.309299 16871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:21.309076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34429 (local address 127.9.66.3:43025)
0925 07:30:21.309221 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:21.309259 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":64,"threads_started":1}
I20260925 07:30:21.309260 16870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:21.308924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34429 (local address 127.9.66.2:44729)
0925 07:30:21.309131 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:21.309169 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":68,"threads_started":1}
I20260925 07:30:21.309655 16872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:21.309416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34429 (local address 127.9.66.1:51643)
0925 07:30:21.309577 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:21.309613 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":49,"threads_started":1}
I20260925 07:30:22.310174 16874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:22.309790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34429 (local address 127.9.66.2:36691)
0925 07:30:22.310066 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:22.310115 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":80,"threads_started":1}
I20260925 07:30:22.310390 16873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:22.309790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34429 (local address 127.9.66.3:38281)
0925 07:30:22.310299 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:22.310339 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":66,"threads_started":1}
I20260925 07:30:22.310653 16875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:22.310103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34429 (local address 127.9.66.1:33105)
0925 07:30:22.310526 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:22.310619 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":48,"threads_started":1}
I20260925 07:30:22.566849 16876 raft_consensus.cc:482] T 00000000000000000000000000000000 P 01d8c4dcbc1244b28cfa2b85723a0a54 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 438b938b95aa4027a1d63ae68aef56ad)
I20260925 07:30:22.566952 16876 raft_consensus.cc:504] T 00000000000000000000000000000000 P 01d8c4dcbc1244b28cfa2b85723a0a54 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "438b938b95aa4027a1d63ae68aef56ad" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 34429 } } peers { permanent_uuid: "01d8c4dcbc1244b28cfa2b85723a0a54" member_type: VOTER last_known_addr { host: "127.9.66.61" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "60014ba2af4c4663ab33a833de5b9113" member_type: NON_VOTER last_known_addr { host: "127.9.66.60" port: 40627 } attrs { promote: true } }
I20260925 07:30:22.567263 16876 leader_election.cc:288] T 00000000000000000000000000000000 P 01d8c4dcbc1244b28cfa2b85723a0a54 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 438b938b95aa4027a1d63ae68aef56ad (127.9.66.62:34429)
I20260925 07:30:22.567643 16877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:22.567423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34429 (local address 127.0.0.1:34482)
0925 07:30:22.567537 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:22.567578 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":74,"threads_started":1}
I20260925 07:30:22.568230 16877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:22.568118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34429 (local address 127.0.0.1:34492)
0925 07:30:22.568163 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:22.568207 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260925 07:30:22.568323 16147 leader_election.cc:334] T 00000000000000000000000000000000 P 01d8c4dcbc1244b28cfa2b85723a0a54 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 438b938b95aa4027a1d63ae68aef56ad (127.9.66.62:34429): Network error: Client connection negotiation failed: client connection to 127.9.66.62:34429: connect: Connection refused (error 111)
I20260925 07:30:22.568382 16147 leader_election.cc:302] T 00000000000000000000000000000000 P 01d8c4dcbc1244b28cfa2b85723a0a54 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 01d8c4dcbc1244b28cfa2b85723a0a54; no voters: 438b938b95aa4027a1d63ae68aef56ad
I20260925 07:30:22.568475 16876 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 01d8c4dcbc1244b28cfa2b85723a0a54 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 07:30:23.311142 16880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:23.310748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34429 (local address 127.9.66.2:51961)
0925 07:30:23.311029 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:23.311078 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":63,"threads_started":1}
I20260925 07:30:23.311275 16879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:23.310740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34429 (local address 127.9.66.3:40507)
0925 07:30:23.311191 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:23.311226 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":60,"threads_started":1}
I20260925 07:30:23.311340 16881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:23.311000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34429 (local address 127.9.66.1:34885)
0925 07:30:23.311269 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:23.311299 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":43,"threads_started":1}
I20260925 07:30:24.312183 16882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:24.311859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34429 (local address 127.9.66.3:46795)
0925 07:30:24.312078 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:24.312120 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":65,"threads_started":1}
I20260925 07:30:24.312188 16884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:24.311979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34429 (local address 127.9.66.2:34449)
0925 07:30:24.312097 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:24.312140 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":52,"threads_started":1}
I20260925 07:30:24.312266 16883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:24.311893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34429 (local address 127.9.66.1:52483)
0925 07:30:24.312196 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:24.312226 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":45,"threads_started":1}
I20260925 07:30:25.312952 16885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:25.312586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34429 (local address 127.9.66.1:42075)
0925 07:30:25.312834 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:25.312886 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":83,"threads_started":1}
I20260925 07:30:25.313169 16886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:25.312776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34429 (local address 127.9.66.2:43905)
0925 07:30:25.313098 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:25.313125 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":65,"threads_started":1}
I20260925 07:30:25.313184 16887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:25.312818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34429 (local address 127.9.66.3:45867)
0925 07:30:25.313089 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:25.313125 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":51,"threads_started":1}
I20260925 07:30:25.323025 16888 raft_consensus.cc:482] T 00000000000000000000000000000000 P 01d8c4dcbc1244b28cfa2b85723a0a54 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 438b938b95aa4027a1d63ae68aef56ad)
I20260925 07:30:25.323097 16888 raft_consensus.cc:504] T 00000000000000000000000000000000 P 01d8c4dcbc1244b28cfa2b85723a0a54 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "438b938b95aa4027a1d63ae68aef56ad" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 34429 } } peers { permanent_uuid: "01d8c4dcbc1244b28cfa2b85723a0a54" member_type: VOTER last_known_addr { host: "127.9.66.61" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "60014ba2af4c4663ab33a833de5b9113" member_type: NON_VOTER last_known_addr { host: "127.9.66.60" port: 40627 } attrs { promote: true } }
I20260925 07:30:25.323249 16888 leader_election.cc:288] T 00000000000000000000000000000000 P 01d8c4dcbc1244b28cfa2b85723a0a54 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 438b938b95aa4027a1d63ae68aef56ad (127.9.66.62:34429)
I20260925 07:30:25.323539 16889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:25.323375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34429 (local address 127.0.0.1:44422)
0925 07:30:25.323471 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:25.323507 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80,"thread_start_us":60,"threads_started":1}
I20260925 07:30:25.323912 16889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:25.323818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34429 (local address 127.0.0.1:44432)
0925 07:30:25.323866 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:25.323890 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260925 07:30:25.323992 16147 leader_election.cc:334] T 00000000000000000000000000000000 P 01d8c4dcbc1244b28cfa2b85723a0a54 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 438b938b95aa4027a1d63ae68aef56ad (127.9.66.62:34429): Network error: Client connection negotiation failed: client connection to 127.9.66.62:34429: connect: Connection refused (error 111)
I20260925 07:30:25.324033 16147 leader_election.cc:302] T 00000000000000000000000000000000 P 01d8c4dcbc1244b28cfa2b85723a0a54 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 01d8c4dcbc1244b28cfa2b85723a0a54; no voters: 438b938b95aa4027a1d63ae68aef56ad
I20260925 07:30:25.324204 16888 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 01d8c4dcbc1244b28cfa2b85723a0a54 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 07:30:26.314030 16893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:26.313837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34429 (local address 127.9.66.1:50151)
0925 07:30:26.313933 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:26.313978 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75,"thread_start_us":45,"threads_started":1}
I20260925 07:30:26.313944 16892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:26.313688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34429 (local address 127.9.66.3:52603)
0925 07:30:26.313830 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:26.313886 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":61,"threads_started":1}
I20260925 07:30:26.314107 16891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:26.313688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34429 (local address 127.9.66.2:49093)
0925 07:30:26.314029 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:26.314063 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":64,"threads_started":1}
I20260925 07:30:27.314795 16896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:27.314584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34429 (local address 127.9.66.2:55509)
0925 07:30:27.314710 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:27.314743 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":60,"threads_started":1}
I20260925 07:30:27.314810 16894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:27.314485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34429 (local address 127.9.66.3:45391)
0925 07:30:27.314711 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:27.314750 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":70,"threads_started":1}
I20260925 07:30:27.314795 16895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:27.314587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34429 (local address 127.9.66.1:46191)
0925 07:30:27.314703 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:27.314743 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":50,"threads_started":1}
I20260925 07:30:28.315474 16898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:28.315205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34429 (local address 127.9.66.2:54533)
0925 07:30:28.315384 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:28.315430 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":63,"threads_started":1}
I20260925 07:30:28.315477 16897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:28.315212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34429 (local address 127.9.66.3:37515)
0925 07:30:28.315385 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:28.315426 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":55,"threads_started":1}
I20260925 07:30:28.315632 16899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:28.315376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34429 (local address 127.9.66.1:39177)
0925 07:30:28.315495 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:28.315523 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":67,"threads_started":1}
I20260925 07:30:29.029762 16900 raft_consensus.cc:482] T 00000000000000000000000000000000 P 01d8c4dcbc1244b28cfa2b85723a0a54 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 438b938b95aa4027a1d63ae68aef56ad)
I20260925 07:30:29.029860 16900 raft_consensus.cc:504] T 00000000000000000000000000000000 P 01d8c4dcbc1244b28cfa2b85723a0a54 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "438b938b95aa4027a1d63ae68aef56ad" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 34429 } } peers { permanent_uuid: "01d8c4dcbc1244b28cfa2b85723a0a54" member_type: VOTER last_known_addr { host: "127.9.66.61" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "60014ba2af4c4663ab33a833de5b9113" member_type: NON_VOTER last_known_addr { host: "127.9.66.60" port: 40627 } attrs { promote: true } }
I20260925 07:30:29.029996 16900 leader_election.cc:288] T 00000000000000000000000000000000 P 01d8c4dcbc1244b28cfa2b85723a0a54 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 438b938b95aa4027a1d63ae68aef56ad (127.9.66.62:34429)
I20260925 07:30:29.030344 16901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:29.030148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34429 (local address 127.0.0.1:44672)
0925 07:30:29.030254 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:29.030301 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":46,"threads_started":1}
W20260925 07:30:29.030480 16147 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.66.62:34429: connect: Connection refused (error 111) [suppressed 2 similar messages]
I20260925 07:30:29.030881 16901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:29.030783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34429 (local address 127.0.0.1:44674)
0925 07:30:29.030831 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:29.030860 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260925 07:30:29.030998 16147 leader_election.cc:334] T 00000000000000000000000000000000 P 01d8c4dcbc1244b28cfa2b85723a0a54 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 438b938b95aa4027a1d63ae68aef56ad (127.9.66.62:34429): Network error: Client connection negotiation failed: client connection to 127.9.66.62:34429: connect: Connection refused (error 111)
I20260925 07:30:29.031047 16147 leader_election.cc:302] T 00000000000000000000000000000000 P 01d8c4dcbc1244b28cfa2b85723a0a54 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 01d8c4dcbc1244b28cfa2b85723a0a54; no voters: 438b938b95aa4027a1d63ae68aef56ad
I20260925 07:30:29.031153 16900 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 01d8c4dcbc1244b28cfa2b85723a0a54 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 07:30:29.316164 16903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:29.315922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34429 (local address 127.9.66.2:60137)
0925 07:30:29.316062 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:29.316102 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":59,"threads_started":1}
I20260925 07:30:29.316223 16904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:29.315905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34429 (local address 127.9.66.3:44131)
0925 07:30:29.316133 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:29.316173 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":74,"threads_started":1}
I20260925 07:30:29.316403 16905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:29.316039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34429 (local address 127.9.66.1:54819)
0925 07:30:29.316295 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:29.316359 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":80,"threads_started":1}
I20260925 07:30:30.316944 16907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:30.316704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34429 (local address 127.9.66.2:49757)
0925 07:30:30.316847 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:30.316886 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":53,"threads_started":1}
I20260925 07:30:30.317042 16906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:30.316650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34429 (local address 127.9.66.3:34839)
0925 07:30:30.316924 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:30.316963 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":79,"threads_started":1}
I20260925 07:30:30.317059 16908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:30.316821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34429 (local address 127.9.66.1:47497)
0925 07:30:30.316985 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:30.317018 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":43,"threads_started":1}
I20260925 07:30:30.789281 16909 raft_consensus.cc:482] T 00000000000000000000000000000000 P 01d8c4dcbc1244b28cfa2b85723a0a54 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 438b938b95aa4027a1d63ae68aef56ad)
I20260925 07:30:30.789420 16909 raft_consensus.cc:504] T 00000000000000000000000000000000 P 01d8c4dcbc1244b28cfa2b85723a0a54 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "438b938b95aa4027a1d63ae68aef56ad" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 34429 } } peers { permanent_uuid: "01d8c4dcbc1244b28cfa2b85723a0a54" member_type: VOTER last_known_addr { host: "127.9.66.61" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "60014ba2af4c4663ab33a833de5b9113" member_type: NON_VOTER last_known_addr { host: "127.9.66.60" port: 40627 } attrs { promote: true } }
I20260925 07:30:30.789628 16909 leader_election.cc:288] T 00000000000000000000000000000000 P 01d8c4dcbc1244b28cfa2b85723a0a54 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 438b938b95aa4027a1d63ae68aef56ad (127.9.66.62:34429)
I20260925 07:30:30.790030 16910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:30.789836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34429 (local address 127.0.0.1:44820)
0925 07:30:30.789947 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:30.789989 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":68,"threads_started":1}
I20260925 07:30:30.790470 16910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:30.790348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34429 (local address 127.0.0.1:44836)
0925 07:30:30.790415 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:30.790447 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260925 07:30:30.790565 16147 leader_election.cc:334] T 00000000000000000000000000000000 P 01d8c4dcbc1244b28cfa2b85723a0a54 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 438b938b95aa4027a1d63ae68aef56ad (127.9.66.62:34429): Network error: Client connection negotiation failed: client connection to 127.9.66.62:34429: connect: Connection refused (error 111)
I20260925 07:30:30.790616 16147 leader_election.cc:302] T 00000000000000000000000000000000 P 01d8c4dcbc1244b28cfa2b85723a0a54 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 01d8c4dcbc1244b28cfa2b85723a0a54; no voters: 438b938b95aa4027a1d63ae68aef56ad
I20260925 07:30:30.790719 16909 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 01d8c4dcbc1244b28cfa2b85723a0a54 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 07:30:31.317783 16913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:31.317344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34429 (local address 127.9.66.1:49735)
0925 07:30:31.317717 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:31.317745 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":64,"threads_started":1}
I20260925 07:30:31.317762 16912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:31.317344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34429 (local address 127.9.66.2:50585)
0925 07:30:31.317641 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:31.317705 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":62,"threads_started":1}
I20260925 07:30:31.317803 16914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:31.317559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34429 (local address 127.9.66.3:59563)
0925 07:30:31.317735 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:31.317766 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":48,"threads_started":1}
I20260925 07:30:32.318518 16915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:32.318270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34429 (local address 127.9.66.2:60177)
0925 07:30:32.318419 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:32.318458 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":85,"threads_started":1}
I20260925 07:30:32.318601 16917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:32.318376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34429 (local address 127.9.66.1:51139)
0925 07:30:32.318527 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:32.318555 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":67,"threads_started":1}
I20260925 07:30:32.318627 16916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:32.318365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34429 (local address 127.9.66.3:60947)
0925 07:30:32.318519 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:32.318555 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":46,"threads_started":1}
I20260925 07:30:33.319224 16918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:33.318894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34429 (local address 127.9.66.2:46783)
0925 07:30:33.319129 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:33.319167 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":62,"threads_started":1}
I20260925 07:30:33.319388 16919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:33.318997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34429 (local address 127.9.66.1:32915)
0925 07:30:33.319319 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:33.319353 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":43,"threads_started":1}
I20260925 07:30:33.319522 16920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:33.319312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34429 (local address 127.9.66.3:47171)
0925 07:30:33.319411 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:33.319472 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":50,"threads_started":1}
I20260925 07:30:34.320184 16922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:34.319940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34429 (local address 127.9.66.2:41219)
0925 07:30:34.320073 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:34.320125 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":42,"threads_started":1}
I20260925 07:30:34.320179 16921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:34.319909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34429 (local address 127.9.66.1:49383)
0925 07:30:34.320089 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:34.320125 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":60,"threads_started":1}
I20260925 07:30:34.320334 16923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:34.320122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34429 (local address 127.9.66.3:56927)
0925 07:30:34.320260 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:34.320290 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":69,"threads_started":1}
I20260925 07:30:35.320915 16924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:35.320588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34429 (local address 127.9.66.1:53307)
0925 07:30:35.320809 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:35.320855 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":66,"threads_started":1}
I20260925 07:30:35.320999 16926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:35.320826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34429 (local address 127.9.66.2:56241)
0925 07:30:35.320920 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:35.320954 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77,"thread_start_us":41,"threads_started":1}
I20260925 07:30:35.321008 16925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:35.320656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34429 (local address 127.9.66.3:51211)
0925 07:30:35.320929 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:35.320958 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":83,"threads_started":1}
I20260925 07:30:36.321834 16927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:36.321481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34429 (local address 127.9.66.1:37197)
0925 07:30:36.321758 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:36.321786 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":178,"threads_started":1}
I20260925 07:30:36.321833 16928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:36.321619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34429 (local address 127.9.66.2:33471)
0925 07:30:36.321741 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:36.321782 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":68,"threads_started":1}
I20260925 07:30:36.321856 16929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:36.321651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34429 (local address 127.9.66.3:57499)
0925 07:30:36.321774 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:36.321808 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":43,"threads_started":1}
I20260925 07:30:37.322649 16930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:37.322430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34429 (local address 127.9.66.3:47921)
0925 07:30:37.322557 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:37.322597 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":61,"threads_started":1}
I20260925 07:30:37.322649 16931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:37.322430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34429 (local address 127.9.66.2:53781)
0925 07:30:37.322557 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:37.322597 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":64,"threads_started":1}
I20260925 07:30:37.322738 16932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:37.322460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34429 (local address 127.9.66.1:35725)
0925 07:30:37.322571 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:37.322673 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":52,"threads_started":1}
I20260925 07:30:38.323385 16933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:38.323097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34429 (local address 127.9.66.2:37241)
0925 07:30:38.323303 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:38.323347 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":65,"threads_started":1}
I20260925 07:30:38.323387 16935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:38.323214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34429 (local address 127.9.66.3:47931)
0925 07:30:38.323320 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:38.323348 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":50,"threads_started":1}
I20260925 07:30:38.323370 16934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:38.323108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34429 (local address 127.9.66.1:56317)
0925 07:30:38.323262 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:38.323308 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":76,"threads_started":1}
I20260925 07:30:39.324119 16936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:39.323866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34429 (local address 127.9.66.2:42721)
0925 07:30:39.324022 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:39.324060 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":80,"threads_started":1}
I20260925 07:30:39.324115 16937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:39.323884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34429 (local address 127.9.66.1:59817)
0925 07:30:39.324014 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:39.324055 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":71,"threads_started":1}
I20260925 07:30:39.324167 16938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:39.324013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34429 (local address 127.9.66.3:39249)
0925 07:30:39.324098 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:39.324125 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73,"thread_start_us":40,"threads_started":1}
I20260925 07:30:39.766862 16939 raft_consensus.cc:482] T 00000000000000000000000000000000 P 01d8c4dcbc1244b28cfa2b85723a0a54 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 438b938b95aa4027a1d63ae68aef56ad)
I20260925 07:30:39.767001 16939 raft_consensus.cc:504] T 00000000000000000000000000000000 P 01d8c4dcbc1244b28cfa2b85723a0a54 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "438b938b95aa4027a1d63ae68aef56ad" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 34429 } } peers { permanent_uuid: "01d8c4dcbc1244b28cfa2b85723a0a54" member_type: VOTER last_known_addr { host: "127.9.66.61" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "60014ba2af4c4663ab33a833de5b9113" member_type: NON_VOTER last_known_addr { host: "127.9.66.60" port: 40627 } attrs { promote: true } }
I20260925 07:30:39.767601 16939 leader_election.cc:288] T 00000000000000000000000000000000 P 01d8c4dcbc1244b28cfa2b85723a0a54 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 438b938b95aa4027a1d63ae68aef56ad (127.9.66.62:34429)
I20260925 07:30:39.767982 16940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:39.767786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34429 (local address 127.0.0.1:40110)
0925 07:30:39.767902 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:39.767946 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":69,"threads_started":1}
W20260925 07:30:39.768064 16147 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.66.62:34429: connect: Connection refused (error 111) [suppressed 1 similar messages]
I20260925 07:30:39.768443 16940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:39.768327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34429 (local address 127.0.0.1:40114)
0925 07:30:39.768379 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:39.768425 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260925 07:30:39.768515 16147 leader_election.cc:334] T 00000000000000000000000000000000 P 01d8c4dcbc1244b28cfa2b85723a0a54 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 438b938b95aa4027a1d63ae68aef56ad (127.9.66.62:34429): Network error: Client connection negotiation failed: client connection to 127.9.66.62:34429: connect: Connection refused (error 111)
I20260925 07:30:39.768558 16147 leader_election.cc:302] T 00000000000000000000000000000000 P 01d8c4dcbc1244b28cfa2b85723a0a54 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 01d8c4dcbc1244b28cfa2b85723a0a54; no voters: 438b938b95aa4027a1d63ae68aef56ad
I20260925 07:30:39.768657 16939 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 01d8c4dcbc1244b28cfa2b85723a0a54 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 07:30:40.324975 16943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:40.324544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34429 (local address 127.9.66.3:49213)
0925 07:30:40.324876 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:40.324921 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":62,"threads_started":1}
I20260925 07:30:40.324970 16944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:40.324758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34429 (local address 127.9.66.1:40021)
0925 07:30:40.324875 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:40.324914 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":57,"threads_started":1}
I20260925 07:30:40.324994 16942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:40.324544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34429 (local address 127.9.66.2:52087)
0925 07:30:40.324928 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:40.324956 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":64,"threads_started":1}
I20260925 07:30:41.325690 16946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:41.325385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34429 (local address 127.9.66.1:53965)
0925 07:30:41.325589 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:41.325631 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":57,"threads_started":1}
I20260925 07:30:41.325767 16945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:41.325339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34429 (local address 127.9.66.3:52577)
0925 07:30:41.325657 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:41.325688 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":61,"threads_started":1}
I20260925 07:30:41.325795 16947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:41.325413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34429 (local address 127.9.66.2:53645)
0925 07:30:41.325703 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:41.325741 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":45,"threads_started":1}
I20260925 07:30:42.326448 16948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:42.326159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34429 (local address 127.9.66.1:43025)
0925 07:30:42.326361 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:42.326398 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":74,"threads_started":1}
I20260925 07:30:42.326460 16949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:42.326226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34429 (local address 127.9.66.3:48443)
0925 07:30:42.326342 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:42.326394 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":43,"threads_started":1}
I20260925 07:30:42.326519 16950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:42.326341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34429 (local address 127.9.66.2:44305)
0925 07:30:42.326450 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:42.326478 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75,"thread_start_us":41,"threads_started":1}
I20260925 07:30:43.327107 16952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:43.326871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34429 (local address 127.9.66.3:33415)
0925 07:30:43.327000 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:43.327045 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":64,"threads_started":1}
I20260925 07:30:43.327121 16951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:43.326870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34429 (local address 127.9.66.1:57511)
0925 07:30:43.327007 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:43.327041 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":66,"threads_started":1}
I20260925 07:30:43.327184 16953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:43.326988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34429 (local address 127.9.66.2:56545)
0925 07:30:43.327102 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:43.327133 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":64,"threads_started":1}
I20260925 07:30:44.327855 16954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:44.327612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34429 (local address 127.9.66.3:54429)
0925 07:30:44.327741 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:44.327790 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":63,"threads_started":1}
I20260925 07:30:44.327852 16955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:44.327612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34429 (local address 127.9.66.1:57815)
0925 07:30:44.327748 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:44.327780 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":65,"threads_started":1}
I20260925 07:30:44.327935 16956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:44.327741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34429 (local address 127.9.66.2:52025)
0925 07:30:44.327859 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:44.327893 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":56,"threads_started":1}
I20260925 07:30:45.328493 16957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:45.328277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34429 (local address 127.9.66.3:34331)
0925 07:30:45.328392 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:45.328436 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":67,"threads_started":1}
I20260925 07:30:45.328518 16958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:45.328273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34429 (local address 127.9.66.1:39645)
0925 07:30:45.328424 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:45.328464 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":90,"threads_started":1}
I20260925 07:30:45.328639 16959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:45.328460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34429 (local address 127.9.66.2:41549)
0925 07:30:45.328556 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:45.328588 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81,"thread_start_us":40,"threads_started":1}
I20260925 07:30:46.329180 16960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:46.328916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34429 (local address 127.9.66.3:34959)
0925 07:30:46.329065 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:46.329113 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":66,"threads_started":1}
I20260925 07:30:46.329263 16962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:46.329070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34429 (local address 127.9.66.1:37343)
0925 07:30:46.329193 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:46.329219 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":73,"threads_started":1}
I20260925 07:30:46.329312 16961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:46.329045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34429 (local address 127.9.66.2:50097)
0925 07:30:46.329182 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:46.329220 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":58,"threads_started":1}
I20260925 07:30:47.329856 16964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:47.329654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34429 (local address 127.9.66.1:33105)
0925 07:30:47.329771 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:47.329805 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":57,"threads_started":1}
I20260925 07:30:47.329856 16965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:47.329654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34429 (local address 127.9.66.2:40221)
0925 07:30:47.329771 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:47.329805 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":55,"threads_started":1}
I20260925 07:30:47.329872 16963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:47.329641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34429 (local address 127.9.66.3:43719)
0925 07:30:47.329788 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:47.329825 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":67,"threads_started":1}
I20260925 07:30:48.330523 16966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:48.330237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34429 (local address 127.9.66.2:59409)
0925 07:30:48.330417 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:48.330462 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":71,"threads_started":1}
I20260925 07:30:48.330605 16967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:48.330316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34429 (local address 127.9.66.1:58093)
0925 07:30:48.330525 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:48.330562 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":69,"threads_started":1}
I20260925 07:30:48.330672 16968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:48.330363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34429 (local address 127.9.66.3:35763)
0925 07:30:48.330579 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:48.330622 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":65,"threads_started":1}
I20260925 07:30:48.347558 16969 raft_consensus.cc:482] T 00000000000000000000000000000000 P 01d8c4dcbc1244b28cfa2b85723a0a54 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 438b938b95aa4027a1d63ae68aef56ad)
I20260925 07:30:48.347656 16969 raft_consensus.cc:504] T 00000000000000000000000000000000 P 01d8c4dcbc1244b28cfa2b85723a0a54 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "438b938b95aa4027a1d63ae68aef56ad" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 34429 } } peers { permanent_uuid: "01d8c4dcbc1244b28cfa2b85723a0a54" member_type: VOTER last_known_addr { host: "127.9.66.61" port: 33901 } attrs { promote: false } } peers { permanent_uuid: "60014ba2af4c4663ab33a833de5b9113" member_type: NON_VOTER last_known_addr { host: "127.9.66.60" port: 40627 } attrs { promote: true } }
I20260925 07:30:48.347782 16969 leader_election.cc:288] T 00000000000000000000000000000000 P 01d8c4dcbc1244b28cfa2b85723a0a54 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 438b938b95aa4027a1d63ae68aef56ad (127.9.66.62:34429)
I20260925 07:30:48.348111 16970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:48.347942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34429 (local address 127.0.0.1:37300)
0925 07:30:48.348042 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:48.348079 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":54,"threads_started":1}
W20260925 07:30:48.348202 16147 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.66.62:34429: connect: Connection refused (error 111)
I20260925 07:30:48.348527 16970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:48.348453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34429 (local address 127.0.0.1:37308)
0925 07:30:48.348489 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:48.348512 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260925 07:30:48.348598 16147 leader_election.cc:334] T 00000000000000000000000000000000 P 01d8c4dcbc1244b28cfa2b85723a0a54 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 438b938b95aa4027a1d63ae68aef56ad (127.9.66.62:34429): Network error: Client connection negotiation failed: client connection to 127.9.66.62:34429: connect: Connection refused (error 111)
I20260925 07:30:48.348686 16147 leader_election.cc:302] T 00000000000000000000000000000000 P 01d8c4dcbc1244b28cfa2b85723a0a54 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 01d8c4dcbc1244b28cfa2b85723a0a54; no voters: 438b938b95aa4027a1d63ae68aef56ad
I20260925 07:30:48.348779 16969 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 01d8c4dcbc1244b28cfa2b85723a0a54 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 07:30:49.331391 16972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:49.331166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34429 (local address 127.9.66.2:54891)
0925 07:30:49.331303 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:49.331340 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":69,"threads_started":1}
I20260925 07:30:49.331398 16973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:49.331166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34429 (local address 127.9.66.3:34971)
0925 07:30:49.331308 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:49.331351 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":66,"threads_started":1}
I20260925 07:30:49.331435 16974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:49.331196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34429 (local address 127.9.66.1:45751)
0925 07:30:49.331303 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:49.331340 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:34429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":62,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:903: Failure
Failed
Bad status: Timed out: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.9.66.62:34429,127.9.66.61:33901,127.9.66.60:40627, num_attempts: 243) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.9.66.62:34429,127.9.66.61:33901,127.9.66.60:40627, num_attempts: 1)
I20260925 07:30:49.504668 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 16308
I20260925 07:30:49.510474 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 16541
I20260925 07:30:49.515491 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 16677
I20260925 07:30:49.519850 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 16027
I20260925 07:30:49.524415 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 16215
2026-09-25T07:30:49Z chronyd exiting
I20260925 07:30:49.540675 9480 test_util.cc:175] -----------------------------------------------
I20260925 07:30:49.540736 9480 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790321392278483-9480-0
[ FAILED ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (33114 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790321392278483-9480-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 07:30:49 dist-test-slave-57fn krb5kdc[16977](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 07:30:49 dist-test-slave-57fn krb5kdc[16977](info): set up 2 sockets
Sep 25 07:30:49 dist-test-slave-57fn krb5kdc[16977](info): commencing operation
krb5kdc: starting...
W20260925 07:30:50.561885 9480 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790321392278483-9480-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790321392278483-9480-0/krb5kdc/test-user.keytab.
Sep 25 07:30:50 dist-test-slave-57fn krb5kdc[16977](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321450, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T07:30:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T07:30:50Z Disabled control of system clock
WARNING: no policy specified for oryx/127.9.66.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.66.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.66.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790321392278483-9480-0/krb5kdc/oryx_127.9.66.62.keytab.
Entry for principal oryx/127.9.66.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790321392278483-9480-0/krb5kdc/oryx_127.9.66.62.keytab.
I20260925 07:30:50.680460 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790321392278483-9480-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790321392278483-9480-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790321392278483-9480-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.66.62:34821
--webserver_interface=127.9.66.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790321392278483-9480-0/krb5kdc/oryx_127.9.66.62.keytab
--principal=oryx/127.9.66.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:45237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.66.62:34821
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:30:50.759415 16990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:30:50.759656 16990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:30:50.759692 16990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:30:50.760826 16990 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:30:50.760890 16990 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 07:30:50.760913 16990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:30:50.760934 16990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:30:50.760960 16990 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:30:50.760982 16990 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:30:50.762524 16990 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:45237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790321392278483-9480-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790321392278483-9480-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.66.62:34821
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790321392278483-9480-0/krb5kdc/oryx_127.9.66.62.keytab
--principal=oryx/127.9.66.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.66.62:34821
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.66.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16990
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790321392278483-9480-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:30:50.762791 16990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:30:50.763007 16990 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 07:30:50.769562 16990 server_base.cc:1034] running on GCE node
W20260925 07:30:50.769565 16995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:30:50.769694 16996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:30:50.769784 16998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:30:50.770063 16990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:30:50.770363 16990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:30:50.771520 16990 hybrid_clock.cc:648] HybridClock initialized: now 1790321450771504 us; error 36 us; skew 500 ppm
Sep 25 07:30:50 dist-test-slave-57fn krb5kdc[16977](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321450, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.66.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:30:50.774112 16990 init.cc:377] Logged in from keytab as oryx/127.9.66.62@KRBTEST.COM (short username oryx)
I20260925 07:30:50.774567 16990 webserver.cc:460] Webserver started at http://127.9.66.62:35071/ using document root <none> and password file <none>
I20260925 07:30:50.774796 16990 fs_manager.cc:359] Metadata directory not provided
I20260925 07:30:50.774860 16990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:30:50.774999 16990 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:30:50.776067 16990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790321392278483-9480-0/minicluster-data/master-0/data/instance:
uuid: "df6bdd3c1abe434eb57ce7f7b79dd302"
format_stamp: "Formatted at 2026-09-25 07:30:50 on dist-test-slave-57fn"
server_key: "df2b5a2cb49bb09dd1021cf8ee683ffd"
server_key_iv: "1dbafd2e86bf8b2145489ef50e384716"
server_key_version: "encryptionkey@0"
I20260925 07:30:50.776432 16990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790321392278483-9480-0/minicluster-data/master-0/wal/instance:
uuid: "df6bdd3c1abe434eb57ce7f7b79dd302"
format_stamp: "Formatted at 2026-09-25 07:30:50 on dist-test-slave-57fn"
server_key: "df2b5a2cb49bb09dd1021cf8ee683ffd"
server_key_iv: "1dbafd2e86bf8b2145489ef50e384716"
server_key_version: "encryptionkey@0"
I20260925 07:30:50.777817 16990 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 07:30:50.778544 17005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:50.778707 16990 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:50.778803 16990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790321392278483-9480-0/minicluster-data/master-0/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790321392278483-9480-0/minicluster-data/master-0/wal
uuid: "df6bdd3c1abe434eb57ce7f7b79dd302"
format_stamp: "Formatted at 2026-09-25 07:30:50 on dist-test-slave-57fn"
server_key: "df2b5a2cb49bb09dd1021cf8ee683ffd"
server_key_iv: "1dbafd2e86bf8b2145489ef50e384716"
server_key_version: "encryptionkey@0"
I20260925 07:30:50.778872 16990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790321392278483-9480-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790321392278483-9480-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790321392278483-9480-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:30:50.797612 16990 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:30:50.799901 16990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:30:50.800067 16990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:30:50.804682 16990 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.62:34821
I20260925 07:30:50.804734 17067 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.62:34821 every 8 connection(s)
I20260925 07:30:50.805043 16990 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
I20260925 07:30:50.805675 17068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:30:50.807972 17068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df6bdd3c1abe434eb57ce7f7b79dd302: Bootstrap starting.
I20260925 07:30:50.808561 17068 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P df6bdd3c1abe434eb57ce7f7b79dd302: Neither blocks nor log segments found. Creating new log.
I20260925 07:30:50.808845 17068 log.cc:824] T 00000000000000000000000000000000 P df6bdd3c1abe434eb57ce7f7b79dd302: Log is configured to *not* fsync() on all Append() calls
I20260925 07:30:50.809474 17068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df6bdd3c1abe434eb57ce7f7b79dd302: No bootstrap required, opened a new log
I20260925 07:30:50.811493 17068 raft_consensus.cc:348] T 00000000000000000000000000000000 P df6bdd3c1abe434eb57ce7f7b79dd302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df6bdd3c1abe434eb57ce7f7b79dd302" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 34821 } }
I20260925 07:30:50.811645 17068 raft_consensus.cc:374] T 00000000000000000000000000000000 P df6bdd3c1abe434eb57ce7f7b79dd302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:30:50.811692 17068 raft_consensus.cc:729] T 00000000000000000000000000000000 P df6bdd3c1abe434eb57ce7f7b79dd302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df6bdd3c1abe434eb57ce7f7b79dd302, State: Initialized, Role: FOLLOWER
I20260925 07:30:50.811839 17068 consensus_queue.cc:260] T 00000000000000000000000000000000 P df6bdd3c1abe434eb57ce7f7b79dd302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df6bdd3c1abe434eb57ce7f7b79dd302" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 34821 } }
I20260925 07:30:50.811915 17068 raft_consensus.cc:388] T 00000000000000000000000000000000 P df6bdd3c1abe434eb57ce7f7b79dd302 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 07:30:50.811966 17068 raft_consensus.cc:482] T 00000000000000000000000000000000 P df6bdd3c1abe434eb57ce7f7b79dd302 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 07:30:50.812016 17068 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df6bdd3c1abe434eb57ce7f7b79dd302 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:30:50.812517 17068 raft_consensus.cc:504] T 00000000000000000000000000000000 P df6bdd3c1abe434eb57ce7f7b79dd302 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df6bdd3c1abe434eb57ce7f7b79dd302" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 34821 } }
I20260925 07:30:50.812628 17068 leader_election.cc:302] T 00000000000000000000000000000000 P df6bdd3c1abe434eb57ce7f7b79dd302 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: df6bdd3c1abe434eb57ce7f7b79dd302; no voters:
I20260925 07:30:50.812795 17068 leader_election.cc:288] T 00000000000000000000000000000000 P df6bdd3c1abe434eb57ce7f7b79dd302 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 07:30:50.812840 17071 raft_consensus.cc:2781] T 00000000000000000000000000000000 P df6bdd3c1abe434eb57ce7f7b79dd302 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:30:50.813035 17071 raft_consensus.cc:686] T 00000000000000000000000000000000 P df6bdd3c1abe434eb57ce7f7b79dd302 [term 1 LEADER]: Becoming Leader. State: Replica: df6bdd3c1abe434eb57ce7f7b79dd302, State: Running, Role: LEADER
I20260925 07:30:50.813050 17068 sys_catalog.cc:564] T 00000000000000000000000000000000 P df6bdd3c1abe434eb57ce7f7b79dd302 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:30:50.813164 17071 consensus_queue.cc:237] T 00000000000000000000000000000000 P df6bdd3c1abe434eb57ce7f7b79dd302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df6bdd3c1abe434eb57ce7f7b79dd302" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 34821 } }
I20260925 07:30:50.813570 17072 sys_catalog.cc:455] T 00000000000000000000000000000000 P df6bdd3c1abe434eb57ce7f7b79dd302 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "df6bdd3c1abe434eb57ce7f7b79dd302" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df6bdd3c1abe434eb57ce7f7b79dd302" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 34821 } } }
I20260925 07:30:50.813781 17072 sys_catalog.cc:458] T 00000000000000000000000000000000 P df6bdd3c1abe434eb57ce7f7b79dd302 [sys.catalog]: This master's current role is: LEADER
I20260925 07:30:50.813710 17073 sys_catalog.cc:455] T 00000000000000000000000000000000 P df6bdd3c1abe434eb57ce7f7b79dd302 [sys.catalog]: SysCatalogTable state changed. Reason: New leader df6bdd3c1abe434eb57ce7f7b79dd302. Latest consensus state: current_term: 1 leader_uuid: "df6bdd3c1abe434eb57ce7f7b79dd302" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df6bdd3c1abe434eb57ce7f7b79dd302" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 34821 } } }
I20260925 07:30:50.813983 17080 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 07:30:50.813990 17073 sys_catalog.cc:458] T 00000000000000000000000000000000 P df6bdd3c1abe434eb57ce7f7b79dd302 [sys.catalog]: This master's current role is: LEADER
I20260925 07:30:50.814518 17080 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 07:30:50.815276 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 16990
I20260925 07:30:50.815385 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790321392278483-9480-0/minicluster-data/master-0/wal/instance
I20260925 07:30:50.815586 9480 external_mini_cluster.cc:1428] Setting key f50170069eb19ab7fb2836d2c44215d7
I20260925 07:30:50.816088 17080 catalog_manager.cc:1347] Generated new cluster ID: 02737b3c06884ff5808a875df95cb018
I20260925 07:30:50.816149 17080 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 07:30:50.824406 17080 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 07:30:50.825062 17080 catalog_manager.cc:1504] Loading token signing keys...
Sep 25 07:30:50 dist-test-slave-57fn krb5kdc[16977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321450, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.66.62@KRBTEST.COM
I20260925 07:30:50.830662 17091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:50.816056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50588 (local address 127.9.66.62:34821)
0925 07:30:50.816228 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:50.816237 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:50.821753 (+ 5516us) server_negotiation.cc:407] Connection header received
0925 07:30:50.821890 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:50.821896 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:50.821947 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:50.822079 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:30:50.822747 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:50.823663 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:50.824603 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:50.825027 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:50.825139 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:50.827492 (+ 2353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:30:50.827495 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:30:50.827496 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:30:50.827510 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:30:50.829135 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:50.829494 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:50.829497 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:50.829497 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:50.829611 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:50.829835 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:50.829836 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:50.829836 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:50.830062 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:30:50.830199 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:50.830303 (+ 104us) server_negotiation.cc:299] Negotiation successful
0925 07:30:50.830453 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":85,"threads_started":1}
I20260925 07:30:50.836730 17080 catalog_manager.cc:5868] T 00000000000000000000000000000000 P df6bdd3c1abe434eb57ce7f7b79dd302: Generated new TSK 0
I20260925 07:30:50.836891 17080 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.9.66.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.66.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.66.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790321392278483-9480-0/krb5kdc/oryx_127.9.66.1.keytab.
Entry for principal oryx/127.9.66.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790321392278483-9480-0/krb5kdc/oryx_127.9.66.1.keytab.
I20260925 07:30:50.869170 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790321392278483-9480-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790321392278483-9480-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790321392278483-9480-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790321392278483-9480-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.66.1:0
--local_ip_for_outbound_sockets=127.9.66.1
--webserver_interface=127.9.66.1
--webserver_port=0
--tserver_master_addrs=127.9.66.62:34821
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790321392278483-9480-0/krb5kdc/oryx_127.9.66.1.keytab
--principal=oryx/127.9.66.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:45237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:30:50.949137 17094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:30:50.949357 17094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:30:50.949394 17094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:30:50.950592 17094 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:30:50.950661 17094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:30:50.950695 17094 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 07:30:50.950737 17094 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.66.1
I20260925 07:30:50.952210 17094 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:45237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790321392278483-9480-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790321392278483-9480-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790321392278483-9480-0/krb5kdc/oryx_127.9.66.1.keytab
--principal=oryx/127.9.66.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.66.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790321392278483-9480-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.66.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.66.62:34821
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17094
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.66.1
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790321392278483-9480-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:30:50.952457 17094 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:30:50.952682 17094 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:30:50.957551 17099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:30:50.957552 17100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:30:50.957580 17102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:30:50.957698 17094 server_base.cc:1034] running on GCE node
I20260925 07:30:50.958016 17094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:30:50.958252 17094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:30:50.959398 17094 hybrid_clock.cc:648] HybridClock initialized: now 1790321450959391 us; error 31 us; skew 500 ppm
Sep 25 07:30:50 dist-test-slave-57fn krb5kdc[16977](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321450, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.66.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:30:50.961755 17094 init.cc:377] Logged in from keytab as oryx/127.9.66.1@KRBTEST.COM (short username oryx)
I20260925 07:30:50.962158 17094 webserver.cc:460] Webserver started at http://127.9.66.1:40831/ using document root <none> and password file <none>
I20260925 07:30:50.962373 17094 fs_manager.cc:359] Metadata directory not provided
I20260925 07:30:50.962435 17094 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:30:50.962576 17094 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:30:50.963572 17094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790321392278483-9480-0/minicluster-data/ts-0/data/instance:
uuid: "d68d1d5e4ad74b64b5e6b439a05d3fd7"
format_stamp: "Formatted at 2026-09-25 07:30:50 on dist-test-slave-57fn"
server_key: "c7460ec94f071e3f432f0b5656c08451"
server_key_iv: "8643ccb6ecca7ea381a1b6cee3eb2ec5"
server_key_version: "encryptionkey@0"
I20260925 07:30:50.963914 17094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790321392278483-9480-0/minicluster-data/ts-0/wal/instance:
uuid: "d68d1d5e4ad74b64b5e6b439a05d3fd7"
format_stamp: "Formatted at 2026-09-25 07:30:50 on dist-test-slave-57fn"
server_key: "c7460ec94f071e3f432f0b5656c08451"
server_key_iv: "8643ccb6ecca7ea381a1b6cee3eb2ec5"
server_key_version: "encryptionkey@0"
I20260925 07:30:50.965214 17094 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 07:30:50.966007 17109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:50.966177 17094 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:50.966267 17094 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790321392278483-9480-0/minicluster-data/ts-0/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790321392278483-9480-0/minicluster-data/ts-0/wal
uuid: "d68d1d5e4ad74b64b5e6b439a05d3fd7"
format_stamp: "Formatted at 2026-09-25 07:30:50 on dist-test-slave-57fn"
server_key: "c7460ec94f071e3f432f0b5656c08451"
server_key_iv: "8643ccb6ecca7ea381a1b6cee3eb2ec5"
server_key_version: "encryptionkey@0"
I20260925 07:30:50.966341 17094 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790321392278483-9480-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790321392278483-9480-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790321392278483-9480-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:30:50.984892 17094 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:30:50.986938 17094 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:30:50.987095 17094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:30:50.987623 17094 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:30:50.987679 17094 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:50.987728 17094 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:30:50.987759 17094 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 07:30:50 dist-test-slave-57fn krb5kdc[16977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321450, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.66.1@KRBTEST.COM for oryx/127.9.66.62@KRBTEST.COM
I20260925 07:30:50.994786 17094 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.1:39919
I20260925 07:30:50.994867 17224 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.1:39919 every 8 connection(s)
I20260925 07:30:50.995196 17094 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790321392278483-9480-0/minicluster-data/ts-0/data/info.pb
I20260925 07:30:50.997668 17091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:50.988169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.1:34895 (local address 127.9.66.62:34821)
0925 07:30:50.988225 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:50.988227 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:50.988435 (+ 208us) server_negotiation.cc:407] Connection header received
0925 07:30:50.989076 (+ 641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:50.989078 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:50.989092 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:50.989126 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:30:50.989965 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:50.990607 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:50.991759 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:50.991974 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:50.992057 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:50.994947 (+ 2890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:30:50.994949 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:30:50.994949 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:30:50.994957 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:30:50.996498 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:50.996845 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:50.996846 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:50.996846 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:50.996895 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:50.997095 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:50.997096 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:50.997096 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:50.997193 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:30:50.997240 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:50.997475 (+ 235us) server_negotiation.cc:299] Negotiation successful
0925 07:30:50.997590 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260925 07:30:50.997895 17118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:50.988161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34821 (local address 127.9.66.1:34895)
0925 07:30:50.988321 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:50.988355 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 07:30:50.988913 (+ 558us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:30:50.989329 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:30:50.989333 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:30:50.989686 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:30:50.989870 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:50.989876 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:50.990704 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:50.990705 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:50.991671 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:50.991674 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:50.992080 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:50.992081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:50.992214 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:50.992736 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:30:50.992744 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:30:50.994856 (+ 2112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:30:50.996564 (+ 1708us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:30:50.996569 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:30:50.996571 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:30:50.996783 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:30:50.996946 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:30:50.996947 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:30:50.996947 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:30:50.997046 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:30:50.997258 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:30:50.997262 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:30:50.997371 (+ 109us) client_negotiation.cc:769] Sending connection context
0925 07:30:50.997453 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 07:30:50.997677 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":82,"threads_started":1}
I20260925 07:30:50.998400 17225 heartbeater.cc:347] Connected to a master server at 127.9.66.62:34821
I20260925 07:30:50.998513 17225 heartbeater.cc:464] Registering TS with master...
I20260925 07:30:50.998760 17225 heartbeater.cc:511] Master 127.9.66.62:34821 requested a full tablet report, sending...
I20260925 07:30:50.999318 17021 ts_manager.cc:194] Registered new tserver with Master: d68d1d5e4ad74b64b5e6b439a05d3fd7 (127.9.66.1:39919)
I20260925 07:30:51.000000 17021 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.66.1@KRBTEST.COM'} at 127.9.66.1:34895
I20260925 07:30:51.004437 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 17094
I20260925 07:30:51.004567 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790321392278483-9480-0/minicluster-data/ts-0/wal/instance
I20260925 07:30:51.004652 17091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:50.999514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.1:36053 (local address 127.9.66.62:34821)
0925 07:30:50.999570 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:50.999572 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:50.999702 (+ 130us) server_negotiation.cc:407] Connection header received
0925 07:30:50.999775 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:50.999776 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:50.999785 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:50.999828 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:30:51.000168 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:51.000635 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:51.001324 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:51.001564 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:51.001631 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:51.002445 (+ 814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:30:51.002446 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:30:51.002446 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:30:51.002453 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:30:51.003552 (+ 1099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:51.003878 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:51.003879 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:51.003879 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:51.003931 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:51.004126 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:51.004127 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:51.004128 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:51.004229 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:30:51.004313 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:51.004467 (+ 154us) server_negotiation.cc:299] Negotiation successful
0925 07:30:51.004547 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 07:30:51.004668 17231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:50.999508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34821 (local address 127.9.66.1:36053)
0925 07:30:50.999634 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:50.999649 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 07:30:50.999717 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:30:50.999830 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:30:50.999830 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:30:51.000008 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:30:51.000075 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:51.000076 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:51.000724 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:51.000725 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:51.001229 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:51.001231 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:51.001638 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:51.001639 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:51.001706 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:51.002086 (+ 380us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:30:51.002089 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:30:51.002350 (+ 261us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:30:51.003636 (+ 1286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:30:51.003638 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:30:51.003638 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:30:51.003799 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:30:51.004000 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:30:51.004000 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:30:51.004001 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:30:51.004055 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:30:51.004317 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:30:51.004319 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:30:51.004361 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 07:30:51.004430 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 07:30:51.004547 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":70,"threads_started":1}
I20260925 07:30:51.004801 9480 external_mini_cluster.cc:1428] Setting key ed6c24e3652d34156905217c7ceaae7b
WARNING: no policy specified for oryx/127.9.66.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.66.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.66.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790321392278483-9480-0/krb5kdc/oryx_127.9.66.2.keytab.
Entry for principal oryx/127.9.66.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790321392278483-9480-0/krb5kdc/oryx_127.9.66.2.keytab.
I20260925 07:30:51.029101 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790321392278483-9480-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790321392278483-9480-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790321392278483-9480-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790321392278483-9480-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.66.2:0
--local_ip_for_outbound_sockets=127.9.66.2
--webserver_interface=127.9.66.2
--webserver_port=0
--tserver_master_addrs=127.9.66.62:34821
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790321392278483-9480-0/krb5kdc/oryx_127.9.66.2.keytab
--principal=oryx/127.9.66.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:45237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:30:51.107825 17234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:30:51.108053 17234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:30:51.108093 17234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:30:51.109210 17234 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:30:51.109292 17234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:30:51.109325 17234 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 07:30:51.109362 17234 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.66.2
I20260925 07:30:51.110853 17234 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:45237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790321392278483-9480-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790321392278483-9480-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790321392278483-9480-0/krb5kdc/oryx_127.9.66.2.keytab
--principal=oryx/127.9.66.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.66.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790321392278483-9480-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.66.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.66.62:34821
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17234
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.66.2
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790321392278483-9480-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:30:51.111114 17234 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:30:51.111330 17234 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 07:30:51.116334 17234 server_base.cc:1034] running on GCE node
W20260925 07:30:51.116350 17240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:30:51.116339 17242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:30:51.116456 17239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:30:51.116683 17234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:30:51.116874 17234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:30:51.118031 17234 hybrid_clock.cc:648] HybridClock initialized: now 1790321451118020 us; error 24 us; skew 500 ppm
Sep 25 07:30:51 dist-test-slave-57fn krb5kdc[16977](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321451, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.66.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:30:51.120445 17234 init.cc:377] Logged in from keytab as oryx/127.9.66.2@KRBTEST.COM (short username oryx)
I20260925 07:30:51.120852 17234 webserver.cc:460] Webserver started at http://127.9.66.2:42631/ using document root <none> and password file <none>
I20260925 07:30:51.121033 17234 fs_manager.cc:359] Metadata directory not provided
I20260925 07:30:51.121071 17234 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:30:51.121150 17234 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:30:51.122233 17234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790321392278483-9480-0/minicluster-data/ts-1/data/instance:
uuid: "64749e2394c84a75819a9f88d38f903a"
format_stamp: "Formatted at 2026-09-25 07:30:51 on dist-test-slave-57fn"
server_key: "b534a1f30267d61065b7f52e96857c46"
server_key_iv: "0dc2f3882f0595a32504a0c17750bf8b"
server_key_version: "encryptionkey@0"
I20260925 07:30:51.122545 17234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790321392278483-9480-0/minicluster-data/ts-1/wal/instance:
uuid: "64749e2394c84a75819a9f88d38f903a"
format_stamp: "Formatted at 2026-09-25 07:30:51 on dist-test-slave-57fn"
server_key: "b534a1f30267d61065b7f52e96857c46"
server_key_iv: "0dc2f3882f0595a32504a0c17750bf8b"
server_key_version: "encryptionkey@0"
I20260925 07:30:51.123790 17234 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 07:30:51.124432 17249 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:51.124570 17234 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 07:30:51.124651 17234 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790321392278483-9480-0/minicluster-data/ts-1/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790321392278483-9480-0/minicluster-data/ts-1/wal
uuid: "64749e2394c84a75819a9f88d38f903a"
format_stamp: "Formatted at 2026-09-25 07:30:51 on dist-test-slave-57fn"
server_key: "b534a1f30267d61065b7f52e96857c46"
server_key_iv: "0dc2f3882f0595a32504a0c17750bf8b"
server_key_version: "encryptionkey@0"
I20260925 07:30:51.124704 17234 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790321392278483-9480-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790321392278483-9480-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790321392278483-9480-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:30:51.138669 17234 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:30:51.140476 17234 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:30:51.140580 17234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:30:51.140998 17234 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:30:51.141063 17234 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:51.141126 17234 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:30:51.141157 17234 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:51.148185 17234 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.2:33491
I20260925 07:30:51.148253 17364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.2:33491 every 8 connection(s)
Sep 25 07:30:51 dist-test-slave-57fn krb5kdc[16977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321451, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.66.2@KRBTEST.COM for oryx/127.9.66.62@KRBTEST.COM
I20260925 07:30:51.148530 17234 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790321392278483-9480-0/minicluster-data/ts-1/data/info.pb
I20260925 07:30:51.151599 17091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:51.141479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.2:60011 (local address 127.9.66.62:34821)
0925 07:30:51.141582 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:51.141585 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:51.141840 (+ 255us) server_negotiation.cc:407] Connection header received
0925 07:30:51.142574 (+ 734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:51.142576 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:51.142586 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:51.142638 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:30:51.143503 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:51.144314 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:51.146019 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:51.146335 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:51.146391 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:51.149092 (+ 2701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:30:51.149093 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:30:51.149094 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:30:51.149101 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:30:51.150452 (+ 1351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:51.150794 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:51.150794 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:51.150795 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:51.150842 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:51.151041 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:51.151042 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:51.151042 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:51.151142 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:30:51.151191 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:51.151476 (+ 285us) server_negotiation.cc:299] Negotiation successful
0925 07:30:51.151538 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 07:30:51.151810 17258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:51.141464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34821 (local address 127.9.66.2:60011)
0925 07:30:51.141695 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:51.141747 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 07:30:51.142462 (+ 715us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:30:51.142737 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:30:51.142742 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:30:51.143124 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:30:51.143417 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:51.143422 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:51.144437 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:51.144438 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:51.145926 (+ 1488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:51.145929 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:51.146419 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:51.146420 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:51.146588 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:51.147146 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:30:51.147166 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:30:51.148985 (+ 1819us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:30:51.150521 (+ 1536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:30:51.150526 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:30:51.150528 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:30:51.150731 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:30:51.150892 (+ 161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:30:51.150892 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:30:51.150893 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:30:51.150992 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:30:51.151219 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:30:51.151225 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:30:51.151346 (+ 121us) client_negotiation.cc:769] Sending connection context
0925 07:30:51.151456 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 07:30:51.151590 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":88,"threads_started":1}
I20260925 07:30:51.152284 17365 heartbeater.cc:347] Connected to a master server at 127.9.66.62:34821
I20260925 07:30:51.152369 17365 heartbeater.cc:464] Registering TS with master...
I20260925 07:30:51.152602 17365 heartbeater.cc:511] Master 127.9.66.62:34821 requested a full tablet report, sending...
I20260925 07:30:51.153031 17021 ts_manager.cc:194] Registered new tserver with Master: 64749e2394c84a75819a9f88d38f903a (127.9.66.2:33491)
I20260925 07:30:51.153594 17021 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.66.2@KRBTEST.COM'} at 127.9.66.2:60011
I20260925 07:30:51.153610 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 17234
I20260925 07:30:51.153723 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790321392278483-9480-0/minicluster-data/ts-1/wal/instance
I20260925 07:30:51.153947 9480 external_mini_cluster.cc:1428] Setting key 9f1e8bd9284dfc3a4f9ddf04bcaf566c
I20260925 07:30:51.159399 17091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:51.153327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.2:45815 (local address 127.9.66.62:34821)
0925 07:30:51.153372 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:51.153374 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:51.153503 (+ 129us) server_negotiation.cc:407] Connection header received
0925 07:30:51.153578 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:51.153580 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:51.153592 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:51.153648 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:30:51.153992 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:51.154673 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:51.155343 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:51.155546 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:51.155623 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:51.156572 (+ 949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:30:51.156574 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:30:51.156574 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:30:51.156583 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:30:51.158203 (+ 1620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:51.158607 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:51.158609 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:51.158609 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:51.158674 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:51.158880 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:51.158880 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:51.158881 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:51.159016 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:30:51.159076 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:51.159271 (+ 195us) server_negotiation.cc:299] Negotiation successful
0925 07:30:51.159336 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 07:30:51.159492 17371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:51.153236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34821 (local address 127.9.66.2:45815)
0925 07:30:51.153429 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:51.153439 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 07:30:51.153497 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:30:51.153674 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:30:51.153675 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:30:51.153872 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:30:51.153918 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:51.153920 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:51.154759 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:51.154760 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:51.155267 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:51.155268 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:51.155635 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:51.155636 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:51.155699 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:51.156166 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:30:51.156171 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:30:51.156483 (+ 312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:30:51.158301 (+ 1818us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:30:51.158302 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:30:51.158303 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:30:51.158539 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:30:51.158746 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:30:51.158746 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:30:51.158747 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:30:51.158816 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:30:51.159087 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:30:51.159088 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:30:51.159138 (+ 50us) client_negotiation.cc:769] Sending connection context
0925 07:30:51.159223 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 07:30:51.159336 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":119,"threads_started":1}
WARNING: no policy specified for oryx/127.9.66.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.66.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.66.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790321392278483-9480-0/krb5kdc/oryx_127.9.66.3.keytab.
Entry for principal oryx/127.9.66.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790321392278483-9480-0/krb5kdc/oryx_127.9.66.3.keytab.
I20260925 07:30:51.181216 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790321392278483-9480-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790321392278483-9480-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790321392278483-9480-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790321392278483-9480-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.66.3:0
--local_ip_for_outbound_sockets=127.9.66.3
--webserver_interface=127.9.66.3
--webserver_port=0
--tserver_master_addrs=127.9.66.62:34821
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790321392278483-9480-0/krb5kdc/oryx_127.9.66.3.keytab
--principal=oryx/127.9.66.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:45237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:30:51.262943 17374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:30:51.263145 17374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:30:51.263170 17374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:30:51.264292 17374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:30:51.264333 17374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:30:51.264353 17374 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 07:30:51.264370 17374 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.66.3
I20260925 07:30:51.265888 17374 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:45237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790321392278483-9480-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790321392278483-9480-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790321392278483-9480-0/krb5kdc/oryx_127.9.66.3.keytab
--principal=oryx/127.9.66.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.66.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790321392278483-9480-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.66.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.66.62:34821
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17374
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.66.3
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790321392278483-9480-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:30:51.266116 17374 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:30:51.266331 17374 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:30:51.271261 17380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:30:51.271252 17382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:30:51.271291 17374 server_base.cc:1034] running on GCE node
W20260925 07:30:51.271302 17379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:30:51.271652 17374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:30:51.271912 17374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:30:51.273075 17374 hybrid_clock.cc:648] HybridClock initialized: now 1790321451273053 us; error 35 us; skew 500 ppm
Sep 25 07:30:51 dist-test-slave-57fn krb5kdc[16977](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321451, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.66.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:30:51.275574 17374 init.cc:377] Logged in from keytab as oryx/127.9.66.3@KRBTEST.COM (short username oryx)
I20260925 07:30:51.276050 17374 webserver.cc:460] Webserver started at http://127.9.66.3:39769/ using document root <none> and password file <none>
I20260925 07:30:51.276283 17374 fs_manager.cc:359] Metadata directory not provided
I20260925 07:30:51.276345 17374 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:30:51.276476 17374 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:30:51.277554 17374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790321392278483-9480-0/minicluster-data/ts-2/data/instance:
uuid: "f19ede8bffa7471e9146e45d3fadd8ce"
format_stamp: "Formatted at 2026-09-25 07:30:51 on dist-test-slave-57fn"
server_key: "e0d794f1328e5628b5a51a804260417b"
server_key_iv: "b9c3be23f0094d43e6894867ff511bda"
server_key_version: "encryptionkey@0"
I20260925 07:30:51.277904 17374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790321392278483-9480-0/minicluster-data/ts-2/wal/instance:
uuid: "f19ede8bffa7471e9146e45d3fadd8ce"
format_stamp: "Formatted at 2026-09-25 07:30:51 on dist-test-slave-57fn"
server_key: "e0d794f1328e5628b5a51a804260417b"
server_key_iv: "b9c3be23f0094d43e6894867ff511bda"
server_key_version: "encryptionkey@0"
I20260925 07:30:51.279160 17374 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:30:51.279847 17389 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:51.280012 17374 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 07:30:51.280103 17374 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790321392278483-9480-0/minicluster-data/ts-2/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790321392278483-9480-0/minicluster-data/ts-2/wal
uuid: "f19ede8bffa7471e9146e45d3fadd8ce"
format_stamp: "Formatted at 2026-09-25 07:30:51 on dist-test-slave-57fn"
server_key: "e0d794f1328e5628b5a51a804260417b"
server_key_iv: "b9c3be23f0094d43e6894867ff511bda"
server_key_version: "encryptionkey@0"
I20260925 07:30:51.280182 17374 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790321392278483-9480-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790321392278483-9480-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790321392278483-9480-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:30:51.309373 17374 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:30:51.311223 17374 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:30:51.311388 17374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:30:51.311890 17374 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:30:51.311952 17374 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:51.311997 17374 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:30:51.312037 17374 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:51.319281 17374 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.3:32865
Sep 25 07:30:51 dist-test-slave-57fn krb5kdc[16977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321451, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.66.3@KRBTEST.COM for oryx/127.9.66.62@KRBTEST.COM
I20260925 07:30:51.319365 17504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.3:32865 every 8 connection(s)
I20260925 07:30:51.319698 17374 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790321392278483-9480-0/minicluster-data/ts-2/data/info.pb
I20260925 07:30:51.322527 17091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:51.312443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.3:57687 (local address 127.9.66.62:34821)
0925 07:30:51.312531 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:51.312535 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:51.312722 (+ 187us) server_negotiation.cc:407] Connection header received
0925 07:30:51.313414 (+ 692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:51.313415 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:51.313427 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:51.313472 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:30:51.314269 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:51.315057 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:51.316664 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:51.317036 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:51.317131 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:51.319832 (+ 2701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:30:51.319833 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:30:51.319834 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:30:51.319842 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:30:51.321164 (+ 1322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:51.321600 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:51.321602 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:51.321602 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:51.321665 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:51.321901 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:51.321902 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:51.321902 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:51.322020 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:30:51.322087 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:51.322359 (+ 272us) server_negotiation.cc:299] Negotiation successful
0925 07:30:51.322443 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 07:30:51.322733 17398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:51.312400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34821 (local address 127.9.66.3:57687)
0925 07:30:51.312579 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:51.312620 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 07:30:51.313314 (+ 694us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:30:51.313589 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:30:51.313594 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:30:51.313910 (+ 316us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:30:51.314139 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:51.314145 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:51.315174 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:51.315176 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:51.316548 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:51.316552 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:51.317198 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:51.317199 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:51.317350 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:51.317867 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:30:51.317878 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:30:51.319729 (+ 1851us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:30:51.321246 (+ 1517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:30:51.321252 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:30:51.321255 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:30:51.321486 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:30:51.321732 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:30:51.321732 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:30:51.321733 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:30:51.321832 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:30:51.322090 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:30:51.322096 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:30:51.322234 (+ 138us) client_negotiation.cc:769] Sending connection context
0925 07:30:51.322324 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 07:30:51.322485 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"spinlock_wait_cycles":18304,"thread_start_us":129,"threads_started":1}
I20260925 07:30:51.323169 17505 heartbeater.cc:347] Connected to a master server at 127.9.66.62:34821
I20260925 07:30:51.323287 17505 heartbeater.cc:464] Registering TS with master...
I20260925 07:30:51.323450 17505 heartbeater.cc:511] Master 127.9.66.62:34821 requested a full tablet report, sending...
I20260925 07:30:51.323846 17022 ts_manager.cc:194] Registered new tserver with Master: f19ede8bffa7471e9146e45d3fadd8ce (127.9.66.3:32865)
I20260925 07:30:51.324330 17022 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.66.3@KRBTEST.COM'} at 127.9.66.3:57687
I20260925 07:30:51.326627 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 17374
I20260925 07:30:51.326747 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790321392278483-9480-0/minicluster-data/ts-2/wal/instance
I20260925 07:30:51.326961 9480 external_mini_cluster.cc:1428] Setting key cafdbedb18a47c029f8f30aa684a6b51
I20260925 07:30:51.327975 9480 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 07:30:51.329793 17091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:51.324162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.3:37089 (local address 127.9.66.62:34821)
0925 07:30:51.324197 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:51.324199 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:51.324318 (+ 119us) server_negotiation.cc:407] Connection header received
0925 07:30:51.324388 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:51.324389 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:51.324398 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:51.324448 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:30:51.324802 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:51.325350 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:51.326038 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:51.326213 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:51.326257 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:51.327126 (+ 869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:30:51.327128 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:30:51.327128 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:30:51.327135 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:30:51.328655 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:51.329041 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:51.329042 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:51.329042 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:51.329090 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:51.329270 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:51.329271 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:51.329271 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:51.329356 (+ 85us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:30:51.329404 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:51.329596 (+ 192us) server_negotiation.cc:299] Negotiation successful
0925 07:30:51.329670 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 07:30:51.329838 17511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:51.324108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34821 (local address 127.9.66.3:37089)
0925 07:30:51.324256 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:51.324265 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 07:30:51.324332 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:30:51.324457 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:30:51.324458 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:30:51.324676 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:30:51.324729 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:51.324731 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:51.325428 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:51.325428 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:51.325967 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:51.325969 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:51.326267 (+ 298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:51.326267 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:51.326309 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:51.326653 (+ 344us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:30:51.326658 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:30:51.327034 (+ 376us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:30:51.328737 (+ 1703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:30:51.328738 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:30:51.328738 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:30:51.328962 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:30:51.329147 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:30:51.329148 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:30:51.329148 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:30:51.329215 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:30:51.329412 (+ 197us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:30:51.329413 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:30:51.329456 (+ 43us) client_negotiation.cc:769] Sending connection context
0925 07:30:51.329573 (+ 117us) client_negotiation.cc:240] Negotiation successful
0925 07:30:51.329692 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":68,"threads_started":1}
Sep 25 07:30:51 dist-test-slave-57fn krb5kdc[16977](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321451, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 07:30:51 dist-test-slave-57fn krb5kdc[16977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.9.66.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 07:30:51 dist-test-slave-57fn krb5kdc[16977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.9.66.62@KRBTEST.COM, Server not found in Kerberos database
W20260925 07:30:51.349479 17091 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:30:51.344242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50592 (local address 127.9.66.62:34821)
0925 07:30:51.344289 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:51.344291 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:51.344300 (+ 9us) server_negotiation.cc:407] Connection header received
0925 07:30:51.344380 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:51.344381 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:51.344391 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:51.344456 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:30:51.345025 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:51.345579 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:51.346339 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:51.346531 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:51.346603 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:51.349378 (+ 2775us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50592: BlockingRecv error: recv got EOF from 127.0.0.1:50592 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
Sep 25 07:30:51 dist-test-slave-57fn krb5kdc[16977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321451, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.9.66.62@KRBTEST.COM
I20260925 07:30:51.356559 17091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:51.350300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50606 (local address 127.9.66.62:34821)
0925 07:30:51.350351 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:51.350353 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:51.350442 (+ 89us) server_negotiation.cc:407] Connection header received
0925 07:30:51.350490 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:51.350491 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:51.350498 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:51.350543 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:30:51.350951 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:51.351383 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:51.352040 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:51.352226 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:51.352275 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:51.354436 (+ 2161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:30:51.354437 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:30:51.354437 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:30:51.354443 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:30:51.355726 (+ 1283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:51.356022 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:51.356022 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:51.356022 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:51.356068 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:51.356216 (+ 148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:51.356216 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:51.356216 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:51.356303 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:30:51.356373 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:51.356451 (+ 78us) server_negotiation.cc:299] Negotiation successful
0925 07:30:51.356501 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260925 07:30:51.357986 17022 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50606:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 07:30:51.359467 17022 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 07:30:51.366118 17528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:51.361823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35462 (local address 127.9.66.3:32865)
0925 07:30:51.362319 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:51.362336 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:51.362346 (+ 10us) server_negotiation.cc:407] Connection header received
0925 07:30:51.362382 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:51.362384 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:51.362399 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:51.362497 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:30:51.362676 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:51.363785 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:51.365153 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:51.365712 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:51.365788 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:51.365864 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:51.365991 (+ 127us) server_negotiation.cc:299] Negotiation successful
0925 07:30:51.366027 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":104,"threads_started":1}
I20260925 07:30:51.366106 17527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:51.361679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.3:32865 (local address 127.0.0.1:35462)
0925 07:30:51.361793 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:51.361830 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 07:30:51.361891 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:30:51.362507 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:30:51.362510 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:30:51.362513 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:30:51.362588 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:51.362590 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:51.363872 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:51.363873 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:51.365076 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:51.365077 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:51.365799 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:51.365800 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:51.365886 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:51.365898 (+ 12us) client_negotiation.cc:769] Sending connection context
0925 07:30:51.365970 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 07:30:51.366023 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84,"mutex_wait_us":4,"thread_start_us":37,"threads_started":1}
I20260925 07:30:51.366210 17529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:51.362107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.2:33491 (local address 127.0.0.1:55282)
0925 07:30:51.362586 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:51.362596 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 07:30:51.362644 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:30:51.362835 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:30:51.362836 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:30:51.362838 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:30:51.362889 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:51.362890 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:51.363709 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:51.363710 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:51.365016 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:51.365018 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:51.365990 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:51.365991 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:51.366070 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:51.366073 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 07:30:51.366115 (+ 42us) client_negotiation.cc:240] Negotiation successful
0925 07:30:51.366150 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"mutex_wait_us":20,"thread_start_us":86,"threads_started":1}
I20260925 07:30:51.366307 17530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:51.362073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50720 (local address 127.9.66.1:39919)
0925 07:30:51.362495 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:51.362499 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:51.362604 (+ 105us) server_negotiation.cc:407] Connection header received
0925 07:30:51.362732 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:51.362735 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:51.362750 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:51.362801 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:30:51.362992 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:51.363972 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:51.364984 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:51.365741 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:51.365811 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:51.365881 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:51.366163 (+ 282us) server_negotiation.cc:299] Negotiation successful
0925 07:30:51.366202 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":100,"threads_started":1}
I20260925 07:30:51.366353 17531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:51.361921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.1:39919 (local address 127.0.0.1:50720)
0925 07:30:51.362539 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:51.362549 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 07:30:51.362640 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:30:51.362883 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:30:51.362883 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:30:51.362885 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:30:51.362920 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:51.362922 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:51.364042 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:51.364043 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:51.364906 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:51.364908 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:51.365916 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:51.365917 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:51.365983 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:51.365990 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 07:30:51.366036 (+ 46us) client_negotiation.cc:240] Negotiation successful
0925 07:30:51.366095 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":602,"mutex_wait_us":1,"thread_start_us":58,"threads_started":1}
I20260925 07:30:51.366521 17532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:51.362269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55282 (local address 127.9.66.2:33491)
0925 07:30:51.362413 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:51.362418 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:51.362701 (+ 283us) server_negotiation.cc:407] Connection header received
0925 07:30:51.362740 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:51.362742 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:51.362761 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:51.362838 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:30:51.362942 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:51.363627 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:51.365101 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:51.365838 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:51.365900 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:51.365962 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:51.366362 (+ 400us) server_negotiation.cc:299] Negotiation successful
0925 07:30:51.366418 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":74,"threads_started":1}
I20260925 07:30:51.367076 17159 tablet_service.cc:1467] Processing CreateTablet for tablet d24fb2b06c8e4455a5e96a0b7a8c2a5e (DEFAULT_TABLE table=test-table [id=810be3e3f7c643a9bf78b9e3243c3646]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:30:51.367144 17439 tablet_service.cc:1467] Processing CreateTablet for tablet d24fb2b06c8e4455a5e96a0b7a8c2a5e (DEFAULT_TABLE table=test-table [id=810be3e3f7c643a9bf78b9e3243c3646]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:30:51.367455 17439 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d24fb2b06c8e4455a5e96a0b7a8c2a5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:30:51.367470 17159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d24fb2b06c8e4455a5e96a0b7a8c2a5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:30:51.371172 17533 tablet_bootstrap.cc:492] T d24fb2b06c8e4455a5e96a0b7a8c2a5e P f19ede8bffa7471e9146e45d3fadd8ce: Bootstrap starting.
I20260925 07:30:51.371184 17534 tablet_bootstrap.cc:492] T d24fb2b06c8e4455a5e96a0b7a8c2a5e P d68d1d5e4ad74b64b5e6b439a05d3fd7: Bootstrap starting.
I20260925 07:30:51.371563 17299 tablet_service.cc:1467] Processing CreateTablet for tablet d24fb2b06c8e4455a5e96a0b7a8c2a5e (DEFAULT_TABLE table=test-table [id=810be3e3f7c643a9bf78b9e3243c3646]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:30:51.371838 17534 tablet_bootstrap.cc:654] T d24fb2b06c8e4455a5e96a0b7a8c2a5e P d68d1d5e4ad74b64b5e6b439a05d3fd7: Neither blocks nor log segments found. Creating new log.
I20260925 07:30:51.371837 17533 tablet_bootstrap.cc:654] T d24fb2b06c8e4455a5e96a0b7a8c2a5e P f19ede8bffa7471e9146e45d3fadd8ce: Neither blocks nor log segments found. Creating new log.
I20260925 07:30:51.371865 17299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d24fb2b06c8e4455a5e96a0b7a8c2a5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:30:51.372189 17534 log.cc:824] T d24fb2b06c8e4455a5e96a0b7a8c2a5e P d68d1d5e4ad74b64b5e6b439a05d3fd7: Log is configured to *not* fsync() on all Append() calls
I20260925 07:30:51.372189 17533 log.cc:824] T d24fb2b06c8e4455a5e96a0b7a8c2a5e P f19ede8bffa7471e9146e45d3fadd8ce: Log is configured to *not* fsync() on all Append() calls
I20260925 07:30:51.372954 17534 tablet_bootstrap.cc:492] T d24fb2b06c8e4455a5e96a0b7a8c2a5e P d68d1d5e4ad74b64b5e6b439a05d3fd7: No bootstrap required, opened a new log
I20260925 07:30:51.373031 17534 ts_tablet_manager.cc:1397] T d24fb2b06c8e4455a5e96a0b7a8c2a5e P d68d1d5e4ad74b64b5e6b439a05d3fd7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 07:30:51.373287 17533 tablet_bootstrap.cc:492] T d24fb2b06c8e4455a5e96a0b7a8c2a5e P f19ede8bffa7471e9146e45d3fadd8ce: No bootstrap required, opened a new log
I20260925 07:30:51.373358 17533 ts_tablet_manager.cc:1397] T d24fb2b06c8e4455a5e96a0b7a8c2a5e P f19ede8bffa7471e9146e45d3fadd8ce: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 07:30:51.374486 17534 raft_consensus.cc:348] T d24fb2b06c8e4455a5e96a0b7a8c2a5e P d68d1d5e4ad74b64b5e6b439a05d3fd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19ede8bffa7471e9146e45d3fadd8ce" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 32865 } } peers { permanent_uuid: "d68d1d5e4ad74b64b5e6b439a05d3fd7" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 39919 } } peers { permanent_uuid: "64749e2394c84a75819a9f88d38f903a" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 33491 } }
I20260925 07:30:51.374639 17534 raft_consensus.cc:374] T d24fb2b06c8e4455a5e96a0b7a8c2a5e P d68d1d5e4ad74b64b5e6b439a05d3fd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:30:51.374680 17534 raft_consensus.cc:729] T d24fb2b06c8e4455a5e96a0b7a8c2a5e P d68d1d5e4ad74b64b5e6b439a05d3fd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d68d1d5e4ad74b64b5e6b439a05d3fd7, State: Initialized, Role: FOLLOWER
I20260925 07:30:51.374817 17534 consensus_queue.cc:260] T d24fb2b06c8e4455a5e96a0b7a8c2a5e P d68d1d5e4ad74b64b5e6b439a05d3fd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19ede8bffa7471e9146e45d3fadd8ce" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 32865 } } peers { permanent_uuid: "d68d1d5e4ad74b64b5e6b439a05d3fd7" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 39919 } } peers { permanent_uuid: "64749e2394c84a75819a9f88d38f903a" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 33491 } }
I20260925 07:30:51.375085 17534 ts_tablet_manager.cc:1428] T d24fb2b06c8e4455a5e96a0b7a8c2a5e P d68d1d5e4ad74b64b5e6b439a05d3fd7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 07:30:51.375102 17225 heartbeater.cc:503] Master 127.9.66.62:34821 was elected leader, sending a full tablet report...
I20260925 07:30:51.375483 17533 raft_consensus.cc:348] T d24fb2b06c8e4455a5e96a0b7a8c2a5e P f19ede8bffa7471e9146e45d3fadd8ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19ede8bffa7471e9146e45d3fadd8ce" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 32865 } } peers { permanent_uuid: "d68d1d5e4ad74b64b5e6b439a05d3fd7" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 39919 } } peers { permanent_uuid: "64749e2394c84a75819a9f88d38f903a" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 33491 } }
I20260925 07:30:51.375638 17533 raft_consensus.cc:374] T d24fb2b06c8e4455a5e96a0b7a8c2a5e P f19ede8bffa7471e9146e45d3fadd8ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:30:51.375670 17533 raft_consensus.cc:729] T d24fb2b06c8e4455a5e96a0b7a8c2a5e P f19ede8bffa7471e9146e45d3fadd8ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f19ede8bffa7471e9146e45d3fadd8ce, State: Initialized, Role: FOLLOWER
I20260925 07:30:51.375798 17533 consensus_queue.cc:260] T d24fb2b06c8e4455a5e96a0b7a8c2a5e P f19ede8bffa7471e9146e45d3fadd8ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19ede8bffa7471e9146e45d3fadd8ce" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 32865 } } peers { permanent_uuid: "d68d1d5e4ad74b64b5e6b439a05d3fd7" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 39919 } } peers { permanent_uuid: "64749e2394c84a75819a9f88d38f903a" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 33491 } }
I20260925 07:30:51.375882 17538 tablet_bootstrap.cc:492] T d24fb2b06c8e4455a5e96a0b7a8c2a5e P 64749e2394c84a75819a9f88d38f903a: Bootstrap starting.
I20260925 07:30:51.376057 17533 ts_tablet_manager.cc:1428] T d24fb2b06c8e4455a5e96a0b7a8c2a5e P f19ede8bffa7471e9146e45d3fadd8ce: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 07:30:51.376163 17505 heartbeater.cc:503] Master 127.9.66.62:34821 was elected leader, sending a full tablet report...
I20260925 07:30:51.376536 17538 tablet_bootstrap.cc:654] T d24fb2b06c8e4455a5e96a0b7a8c2a5e P 64749e2394c84a75819a9f88d38f903a: Neither blocks nor log segments found. Creating new log.
I20260925 07:30:51.376820 17538 log.cc:824] T d24fb2b06c8e4455a5e96a0b7a8c2a5e P 64749e2394c84a75819a9f88d38f903a: Log is configured to *not* fsync() on all Append() calls
I20260925 07:30:51.377580 17538 tablet_bootstrap.cc:492] T d24fb2b06c8e4455a5e96a0b7a8c2a5e P 64749e2394c84a75819a9f88d38f903a: No bootstrap required, opened a new log
I20260925 07:30:51.377660 17538 ts_tablet_manager.cc:1397] T d24fb2b06c8e4455a5e96a0b7a8c2a5e P 64749e2394c84a75819a9f88d38f903a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 07:30:51.378934 17538 raft_consensus.cc:348] T d24fb2b06c8e4455a5e96a0b7a8c2a5e P 64749e2394c84a75819a9f88d38f903a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19ede8bffa7471e9146e45d3fadd8ce" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 32865 } } peers { permanent_uuid: "d68d1d5e4ad74b64b5e6b439a05d3fd7" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 39919 } } peers { permanent_uuid: "64749e2394c84a75819a9f88d38f903a" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 33491 } }
I20260925 07:30:51.379070 17538 raft_consensus.cc:374] T d24fb2b06c8e4455a5e96a0b7a8c2a5e P 64749e2394c84a75819a9f88d38f903a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:30:51.379125 17538 raft_consensus.cc:729] T d24fb2b06c8e4455a5e96a0b7a8c2a5e P 64749e2394c84a75819a9f88d38f903a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64749e2394c84a75819a9f88d38f903a, State: Initialized, Role: FOLLOWER
I20260925 07:30:51.379231 17538 consensus_queue.cc:260] T d24fb2b06c8e4455a5e96a0b7a8c2a5e P 64749e2394c84a75819a9f88d38f903a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19ede8bffa7471e9146e45d3fadd8ce" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 32865 } } peers { permanent_uuid: "d68d1d5e4ad74b64b5e6b439a05d3fd7" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 39919 } } peers { permanent_uuid: "64749e2394c84a75819a9f88d38f903a" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 33491 } }
I20260925 07:30:51.379452 17538 ts_tablet_manager.cc:1428] T d24fb2b06c8e4455a5e96a0b7a8c2a5e P 64749e2394c84a75819a9f88d38f903a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 07:30:51.379508 17365 heartbeater.cc:503] Master 127.9.66.62:34821 was elected leader, sending a full tablet report...
W20260925 07:30:51.399044 17366 tablet.cc:2369] T d24fb2b06c8e4455a5e96a0b7a8c2a5e P 64749e2394c84a75819a9f88d38f903a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 07:30:51.496279 17226 tablet.cc:2369] T d24fb2b06c8e4455a5e96a0b7a8c2a5e P d68d1d5e4ad74b64b5e6b439a05d3fd7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 07:30:51.556506 17539 raft_consensus.cc:482] T d24fb2b06c8e4455a5e96a0b7a8c2a5e P f19ede8bffa7471e9146e45d3fadd8ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 07:30:51.556658 17539 raft_consensus.cc:504] T d24fb2b06c8e4455a5e96a0b7a8c2a5e P f19ede8bffa7471e9146e45d3fadd8ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19ede8bffa7471e9146e45d3fadd8ce" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 32865 } } peers { permanent_uuid: "d68d1d5e4ad74b64b5e6b439a05d3fd7" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 39919 } } peers { permanent_uuid: "64749e2394c84a75819a9f88d38f903a" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 33491 } }
I20260925 07:30:51.557027 17539 leader_election.cc:288] T d24fb2b06c8e4455a5e96a0b7a8c2a5e P f19ede8bffa7471e9146e45d3fadd8ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d68d1d5e4ad74b64b5e6b439a05d3fd7 (127.9.66.1:39919), 64749e2394c84a75819a9f88d38f903a (127.9.66.2:33491)
I20260925 07:30:51.560518 17532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:51.557530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.3:37385 (local address 127.9.66.2:33491)
0925 07:30:51.557579 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:51.557581 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:51.557662 (+ 81us) server_negotiation.cc:407] Connection header received
0925 07:30:51.557721 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:51.557722 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:51.557736 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:51.557801 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:30:51.558002 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:51.558486 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:51.559709 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:51.560109 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:51.560167 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:51.560177 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:51.560370 (+ 193us) server_negotiation.cc:299] Negotiation successful
0925 07:30:51.560446 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 07:30:51.560546 17398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:51.557225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.1:39919 (local address 127.9.66.3:57667)
0925 07:30:51.557304 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:51.557316 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 07:30:51.557398 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:30:51.557659 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:30:51.557660 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:30:51.557662 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:30:51.557751 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:51.557753 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:51.558664 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:51.558665 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:51.559563 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:51.559566 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:51.560263 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:51.560264 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:51.560319 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:51.560322 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 07:30:51.560393 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 07:30:51.560432 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50,"mutex_wait_us":34}
I20260925 07:30:51.560559 17530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:51.557313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.3:57667 (local address 127.9.66.1:39919)
0925 07:30:51.557377 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:51.557379 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:51.557392 (+ 13us) server_negotiation.cc:407] Connection header received
0925 07:30:51.557542 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:51.557544 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:51.557559 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:51.557632 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:30:51.557831 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:51.558458 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:51.559776 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:51.560128 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:51.560258 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:51.560269 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:51.560445 (+ 176us) server_negotiation.cc:299] Negotiation successful
0925 07:30:51.560492 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 07:30:51.560568 17542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:51.557373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.2:33491 (local address 127.9.66.3:37385)
0925 07:30:51.557594 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:51.557604 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 07:30:51.557657 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:30:51.557826 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:30:51.557827 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:30:51.557828 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:30:51.557902 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:51.557904 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:51.558710 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:51.558711 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:51.559551 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:51.559553 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:51.560209 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:51.560210 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:51.560290 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:51.560295 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 07:30:51.560353 (+ 58us) client_negotiation.cc:240] Negotiation successful
0925 07:30:51.560410 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"mutex_wait_us":31,"thread_start_us":122,"threads_started":1}
I20260925 07:30:51.560992 17319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24fb2b06c8e4455a5e96a0b7a8c2a5e" candidate_uuid: "f19ede8bffa7471e9146e45d3fadd8ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64749e2394c84a75819a9f88d38f903a" is_pre_election: true
I20260925 07:30:51.561038 17179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24fb2b06c8e4455a5e96a0b7a8c2a5e" candidate_uuid: "f19ede8bffa7471e9146e45d3fadd8ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d68d1d5e4ad74b64b5e6b439a05d3fd7" is_pre_election: true
I20260925 07:30:51.561156 17319 raft_consensus.cc:2454] T d24fb2b06c8e4455a5e96a0b7a8c2a5e P 64749e2394c84a75819a9f88d38f903a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f19ede8bffa7471e9146e45d3fadd8ce in term 0.
I20260925 07:30:51.561187 17179 raft_consensus.cc:2454] T d24fb2b06c8e4455a5e96a0b7a8c2a5e P d68d1d5e4ad74b64b5e6b439a05d3fd7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f19ede8bffa7471e9146e45d3fadd8ce in term 0.
I20260925 07:30:51.561378 17391 leader_election.cc:302] T d24fb2b06c8e4455a5e96a0b7a8c2a5e P f19ede8bffa7471e9146e45d3fadd8ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64749e2394c84a75819a9f88d38f903a, f19ede8bffa7471e9146e45d3fadd8ce; no voters:
I20260925 07:30:51.561611 17539 raft_consensus.cc:2781] T d24fb2b06c8e4455a5e96a0b7a8c2a5e P f19ede8bffa7471e9146e45d3fadd8ce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 07:30:51.561690 17539 raft_consensus.cc:482] T d24fb2b06c8e4455a5e96a0b7a8c2a5e P f19ede8bffa7471e9146e45d3fadd8ce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 07:30:51.561709 17539 raft_consensus.cc:3037] T d24fb2b06c8e4455a5e96a0b7a8c2a5e P f19ede8bffa7471e9146e45d3fadd8ce [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:30:51.562695 17539 raft_consensus.cc:504] T d24fb2b06c8e4455a5e96a0b7a8c2a5e P f19ede8bffa7471e9146e45d3fadd8ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19ede8bffa7471e9146e45d3fadd8ce" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 32865 } } peers { permanent_uuid: "d68d1d5e4ad74b64b5e6b439a05d3fd7" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 39919 } } peers { permanent_uuid: "64749e2394c84a75819a9f88d38f903a" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 33491 } }
I20260925 07:30:51.562804 17539 leader_election.cc:288] T d24fb2b06c8e4455a5e96a0b7a8c2a5e P f19ede8bffa7471e9146e45d3fadd8ce [CANDIDATE]: Term 1 election: Requested vote from peers d68d1d5e4ad74b64b5e6b439a05d3fd7 (127.9.66.1:39919), 64749e2394c84a75819a9f88d38f903a (127.9.66.2:33491)
I20260925 07:30:51.563050 17179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24fb2b06c8e4455a5e96a0b7a8c2a5e" candidate_uuid: "f19ede8bffa7471e9146e45d3fadd8ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d68d1d5e4ad74b64b5e6b439a05d3fd7"
I20260925 07:30:51.563098 17319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24fb2b06c8e4455a5e96a0b7a8c2a5e" candidate_uuid: "f19ede8bffa7471e9146e45d3fadd8ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64749e2394c84a75819a9f88d38f903a"
I20260925 07:30:51.563134 17179 raft_consensus.cc:3037] T d24fb2b06c8e4455a5e96a0b7a8c2a5e P d68d1d5e4ad74b64b5e6b439a05d3fd7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:30:51.563202 17319 raft_consensus.cc:3037] T d24fb2b06c8e4455a5e96a0b7a8c2a5e P 64749e2394c84a75819a9f88d38f903a [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:30:51.563807 17179 raft_consensus.cc:2454] T d24fb2b06c8e4455a5e96a0b7a8c2a5e P d68d1d5e4ad74b64b5e6b439a05d3fd7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f19ede8bffa7471e9146e45d3fadd8ce in term 1.
I20260925 07:30:51.563985 17391 leader_election.cc:302] T d24fb2b06c8e4455a5e96a0b7a8c2a5e P f19ede8bffa7471e9146e45d3fadd8ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d68d1d5e4ad74b64b5e6b439a05d3fd7, f19ede8bffa7471e9146e45d3fadd8ce; no voters:
I20260925 07:30:51.564023 17319 raft_consensus.cc:2454] T d24fb2b06c8e4455a5e96a0b7a8c2a5e P 64749e2394c84a75819a9f88d38f903a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f19ede8bffa7471e9146e45d3fadd8ce in term 1.
I20260925 07:30:51.564095 17539 raft_consensus.cc:2781] T d24fb2b06c8e4455a5e96a0b7a8c2a5e P f19ede8bffa7471e9146e45d3fadd8ce [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:30:51.564263 17539 raft_consensus.cc:686] T d24fb2b06c8e4455a5e96a0b7a8c2a5e P f19ede8bffa7471e9146e45d3fadd8ce [term 1 LEADER]: Becoming Leader. State: Replica: f19ede8bffa7471e9146e45d3fadd8ce, State: Running, Role: LEADER
I20260925 07:30:51.564366 17539 consensus_queue.cc:237] T d24fb2b06c8e4455a5e96a0b7a8c2a5e P f19ede8bffa7471e9146e45d3fadd8ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19ede8bffa7471e9146e45d3fadd8ce" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 32865 } } peers { permanent_uuid: "d68d1d5e4ad74b64b5e6b439a05d3fd7" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 39919 } } peers { permanent_uuid: "64749e2394c84a75819a9f88d38f903a" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 33491 } }
I20260925 07:30:51.565220 17022 catalog_manager.cc:5495] T d24fb2b06c8e4455a5e96a0b7a8c2a5e P f19ede8bffa7471e9146e45d3fadd8ce reported cstate change: term changed from 0 to 1, leader changed from <none> to f19ede8bffa7471e9146e45d3fadd8ce (127.9.66.3). New cstate: current_term: 1 leader_uuid: "f19ede8bffa7471e9146e45d3fadd8ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f19ede8bffa7471e9146e45d3fadd8ce" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 32865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d68d1d5e4ad74b64b5e6b439a05d3fd7" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 39919 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64749e2394c84a75819a9f88d38f903a" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 33491 } health_report { overall_health: UNKNOWN } } }
W20260925 07:30:51.570232 17506 tablet.cc:2369] T d24fb2b06c8e4455a5e96a0b7a8c2a5e P f19ede8bffa7471e9146e45d3fadd8ce: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 07:30:51.590775 17528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:51.588263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35472 (local address 127.9.66.3:32865)
0925 07:30:51.588310 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:51.588312 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:51.588326 (+ 14us) server_negotiation.cc:407] Connection header received
0925 07:30:51.588402 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:51.588402 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:51.588420 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:51.588500 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 07:30:51.588662 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:51.589129 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:51.589945 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:51.590139 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:51.590222 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:51.590422 (+ 200us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 07:30:51.590559 (+ 137us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 07:30:51.590651 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:51.590670 (+ 19us) server_negotiation.cc:299] Negotiation successful
0925 07:30:51.590688 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 07:30:51.592504 17319 raft_consensus.cc:1261] T d24fb2b06c8e4455a5e96a0b7a8c2a5e P 64749e2394c84a75819a9f88d38f903a [term 1 FOLLOWER]: Refusing update from remote peer f19ede8bffa7471e9146e45d3fadd8ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:30:51.592509 17179 raft_consensus.cc:1261] T d24fb2b06c8e4455a5e96a0b7a8c2a5e P d68d1d5e4ad74b64b5e6b439a05d3fd7 [term 1 FOLLOWER]: Refusing update from remote peer f19ede8bffa7471e9146e45d3fadd8ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:30:51.592826 17543 consensus_queue.cc:1035] T d24fb2b06c8e4455a5e96a0b7a8c2a5e P f19ede8bffa7471e9146e45d3fadd8ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "d68d1d5e4ad74b64b5e6b439a05d3fd7" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 39919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:30:51.592947 17539 consensus_queue.cc:1035] T d24fb2b06c8e4455a5e96a0b7a8c2a5e P f19ede8bffa7471e9146e45d3fadd8ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "64749e2394c84a75819a9f88d38f903a" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 33491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:30:51.595661 17548 mvcc.cc:205] Tried to move back new op lower bound from 7333156665720000512 to 7333156665608355840. Current Snapshot: MvccSnapshot[applied={T|T < 7333156665720000512}]
I20260925 07:30:51.596299 17547 mvcc.cc:205] Tried to move back new op lower bound from 7333156665720000512 to 7333156665608355840. Current Snapshot: MvccSnapshot[applied={T|T < 7333156665720000512}]
I20260925 07:30:51.597410 17021 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50606:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 07:30:51.597499 17021 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50606:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 07:30:51.598222 17021 catalog_manager.cc:5782] T 00000000000000000000000000000000 P df6bdd3c1abe434eb57ce7f7b79dd302: Sending DeleteTablet for 3 replicas of tablet d24fb2b06c8e4455a5e96a0b7a8c2a5e
I20260925 07:30:51.598515 17439 tablet_service.cc:1514] Processing DeleteTablet for tablet d24fb2b06c8e4455a5e96a0b7a8c2a5e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:30:51 UTC) from {username='oryx', principal='oryx/127.9.66.62@KRBTEST.COM'} at 127.0.0.1:35462
I20260925 07:30:51.598635 17159 tablet_service.cc:1514] Processing DeleteTablet for tablet d24fb2b06c8e4455a5e96a0b7a8c2a5e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:30:51 UTC) from {username='oryx', principal='oryx/127.9.66.62@KRBTEST.COM'} at 127.0.0.1:50720
I20260925 07:30:51.598729 17555 tablet_replica.cc:330] T d24fb2b06c8e4455a5e96a0b7a8c2a5e P f19ede8bffa7471e9146e45d3fadd8ce: stopping tablet replica
I20260925 07:30:51.598757 17299 tablet_service.cc:1514] Processing DeleteTablet for tablet d24fb2b06c8e4455a5e96a0b7a8c2a5e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:30:51 UTC) from {username='oryx', principal='oryx/127.9.66.62@KRBTEST.COM'} at 127.0.0.1:55282
I20260925 07:30:51.598814 17556 tablet_replica.cc:330] T d24fb2b06c8e4455a5e96a0b7a8c2a5e P d68d1d5e4ad74b64b5e6b439a05d3fd7: stopping tablet replica
I20260925 07:30:51.598819 17555 raft_consensus.cc:2229] T d24fb2b06c8e4455a5e96a0b7a8c2a5e P f19ede8bffa7471e9146e45d3fadd8ce [term 1 LEADER]: Raft consensus shutting down.
I20260925 07:30:51.598881 17556 raft_consensus.cc:2229] T d24fb2b06c8e4455a5e96a0b7a8c2a5e P d68d1d5e4ad74b64b5e6b439a05d3fd7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 07:30:51.598924 17555 raft_consensus.cc:2258] T d24fb2b06c8e4455a5e96a0b7a8c2a5e P f19ede8bffa7471e9146e45d3fadd8ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:30:51.598954 17556 raft_consensus.cc:2258] T d24fb2b06c8e4455a5e96a0b7a8c2a5e P d68d1d5e4ad74b64b5e6b439a05d3fd7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:30:51.599020 17021 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50606:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 07:30:51.599157 17021 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 07:30:51.599433 17555 ts_tablet_manager.cc:1905] T d24fb2b06c8e4455a5e96a0b7a8c2a5e P f19ede8bffa7471e9146e45d3fadd8ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 07:30:51.599434 17556 ts_tablet_manager.cc:1905] T d24fb2b06c8e4455a5e96a0b7a8c2a5e P d68d1d5e4ad74b64b5e6b439a05d3fd7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 07:30:51.600788 17159 tablet_service.cc:1467] Processing CreateTablet for tablet 8f24bd02d9234148a267c21f227d833c (DEFAULT_TABLE table=test-table [id=5c67774f8c1e494182a71992d4375c29]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:30:51.600965 17439 tablet_service.cc:1467] Processing CreateTablet for tablet 8f24bd02d9234148a267c21f227d833c (DEFAULT_TABLE table=test-table [id=5c67774f8c1e494182a71992d4375c29]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:30:51.601117 17159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f24bd02d9234148a267c21f227d833c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:30:51.601107 17439 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f24bd02d9234148a267c21f227d833c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:30:51.601274 17299 tablet_service.cc:1467] Processing CreateTablet for tablet 8f24bd02d9234148a267c21f227d833c (DEFAULT_TABLE table=test-table [id=5c67774f8c1e494182a71992d4375c29]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:30:51.601367 17299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f24bd02d9234148a267c21f227d833c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:30:51.602267 17534 tablet_bootstrap.cc:492] T 8f24bd02d9234148a267c21f227d833c P d68d1d5e4ad74b64b5e6b439a05d3fd7: Bootstrap starting.
I20260925 07:30:51.602306 17533 tablet_bootstrap.cc:492] T 8f24bd02d9234148a267c21f227d833c P f19ede8bffa7471e9146e45d3fadd8ce: Bootstrap starting.
I20260925 07:30:51.602684 17534 tablet_bootstrap.cc:654] T 8f24bd02d9234148a267c21f227d833c P d68d1d5e4ad74b64b5e6b439a05d3fd7: Neither blocks nor log segments found. Creating new log.
I20260925 07:30:51.602705 17538 tablet_bootstrap.cc:492] T 8f24bd02d9234148a267c21f227d833c P 64749e2394c84a75819a9f88d38f903a: Bootstrap starting.
I20260925 07:30:51.602712 17533 tablet_bootstrap.cc:654] T 8f24bd02d9234148a267c21f227d833c P f19ede8bffa7471e9146e45d3fadd8ce: Neither blocks nor log segments found. Creating new log.
I20260925 07:30:51.603111 17538 tablet_bootstrap.cc:654] T 8f24bd02d9234148a267c21f227d833c P 64749e2394c84a75819a9f88d38f903a: Neither blocks nor log segments found. Creating new log.
I20260925 07:30:51.603145 17556 ts_tablet_manager.cc:1918] T d24fb2b06c8e4455a5e96a0b7a8c2a5e P d68d1d5e4ad74b64b5e6b439a05d3fd7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 07:30:51.603210 17556 log.cc:1196] T d24fb2b06c8e4455a5e96a0b7a8c2a5e P d68d1d5e4ad74b64b5e6b439a05d3fd7: Deleting WAL directory at /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790321392278483-9480-0/minicluster-data/ts-0/wal/wals/d24fb2b06c8e4455a5e96a0b7a8c2a5e
I20260925 07:30:51.601105 17555 ts_tablet_manager.cc:1918] T d24fb2b06c8e4455a5e96a0b7a8c2a5e P f19ede8bffa7471e9146e45d3fadd8ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 07:30:51.603327 17555 log.cc:1196] T d24fb2b06c8e4455a5e96a0b7a8c2a5e P f19ede8bffa7471e9146e45d3fadd8ce: Deleting WAL directory at /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790321392278483-9480-0/minicluster-data/ts-2/wal/wals/d24fb2b06c8e4455a5e96a0b7a8c2a5e
I20260925 07:30:51.603468 17556 ts_tablet_manager.cc:1939] T d24fb2b06c8e4455a5e96a0b7a8c2a5e P d68d1d5e4ad74b64b5e6b439a05d3fd7: Deleting consensus metadata
I20260925 07:30:51.603559 17555 ts_tablet_manager.cc:1939] T d24fb2b06c8e4455a5e96a0b7a8c2a5e P f19ede8bffa7471e9146e45d3fadd8ce: Deleting consensus metadata
I20260925 07:30:51.603832 17007 catalog_manager.cc:4841] TS d68d1d5e4ad74b64b5e6b439a05d3fd7 (127.9.66.1:39919): tablet d24fb2b06c8e4455a5e96a0b7a8c2a5e (table test-table [id=810be3e3f7c643a9bf78b9e3243c3646]) successfully deleted
I20260925 07:30:51.603909 17007 catalog_manager.cc:4841] TS f19ede8bffa7471e9146e45d3fadd8ce (127.9.66.3:32865): tablet d24fb2b06c8e4455a5e96a0b7a8c2a5e (table test-table [id=810be3e3f7c643a9bf78b9e3243c3646]) successfully deleted
I20260925 07:30:51.603778 17557 tablet_replica.cc:330] T d24fb2b06c8e4455a5e96a0b7a8c2a5e P 64749e2394c84a75819a9f88d38f903a: stopping tablet replica
I20260925 07:30:51.604108 17557 raft_consensus.cc:2229] T d24fb2b06c8e4455a5e96a0b7a8c2a5e P 64749e2394c84a75819a9f88d38f903a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 07:30:51.604228 17533 tablet_bootstrap.cc:492] T 8f24bd02d9234148a267c21f227d833c P f19ede8bffa7471e9146e45d3fadd8ce: No bootstrap required, opened a new log
I20260925 07:30:51.604305 17533 ts_tablet_manager.cc:1397] T 8f24bd02d9234148a267c21f227d833c P f19ede8bffa7471e9146e45d3fadd8ce: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 07:30:51.604332 17538 tablet_bootstrap.cc:492] T 8f24bd02d9234148a267c21f227d833c P 64749e2394c84a75819a9f88d38f903a: No bootstrap required, opened a new log
I20260925 07:30:51.604386 17538 ts_tablet_manager.cc:1397] T 8f24bd02d9234148a267c21f227d833c P 64749e2394c84a75819a9f88d38f903a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 07:30:51.604437 17533 raft_consensus.cc:348] T 8f24bd02d9234148a267c21f227d833c P f19ede8bffa7471e9146e45d3fadd8ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d68d1d5e4ad74b64b5e6b439a05d3fd7" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 39919 } } peers { permanent_uuid: "64749e2394c84a75819a9f88d38f903a" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 33491 } } peers { permanent_uuid: "f19ede8bffa7471e9146e45d3fadd8ce" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 32865 } }
I20260925 07:30:51.604488 17533 raft_consensus.cc:374] T 8f24bd02d9234148a267c21f227d833c P f19ede8bffa7471e9146e45d3fadd8ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:30:51.604501 17533 raft_consensus.cc:729] T 8f24bd02d9234148a267c21f227d833c P f19ede8bffa7471e9146e45d3fadd8ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f19ede8bffa7471e9146e45d3fadd8ce, State: Initialized, Role: FOLLOWER
I20260925 07:30:51.604534 17538 raft_consensus.cc:348] T 8f24bd02d9234148a267c21f227d833c P 64749e2394c84a75819a9f88d38f903a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d68d1d5e4ad74b64b5e6b439a05d3fd7" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 39919 } } peers { permanent_uuid: "64749e2394c84a75819a9f88d38f903a" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 33491 } } peers { permanent_uuid: "f19ede8bffa7471e9146e45d3fadd8ce" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 32865 } }
I20260925 07:30:51.604552 17533 consensus_queue.cc:260] T 8f24bd02d9234148a267c21f227d833c P f19ede8bffa7471e9146e45d3fadd8ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d68d1d5e4ad74b64b5e6b439a05d3fd7" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 39919 } } peers { permanent_uuid: "64749e2394c84a75819a9f88d38f903a" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 33491 } } peers { permanent_uuid: "f19ede8bffa7471e9146e45d3fadd8ce" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 32865 } }
I20260925 07:30:51.604595 17538 raft_consensus.cc:374] T 8f24bd02d9234148a267c21f227d833c P 64749e2394c84a75819a9f88d38f903a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:30:51.604616 17538 raft_consensus.cc:729] T 8f24bd02d9234148a267c21f227d833c P 64749e2394c84a75819a9f88d38f903a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64749e2394c84a75819a9f88d38f903a, State: Initialized, Role: FOLLOWER
I20260925 07:30:51.604636 17533 ts_tablet_manager.cc:1428] T 8f24bd02d9234148a267c21f227d833c P f19ede8bffa7471e9146e45d3fadd8ce: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:51.604655 17538 consensus_queue.cc:260] T 8f24bd02d9234148a267c21f227d833c P 64749e2394c84a75819a9f88d38f903a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d68d1d5e4ad74b64b5e6b439a05d3fd7" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 39919 } } peers { permanent_uuid: "64749e2394c84a75819a9f88d38f903a" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 33491 } } peers { permanent_uuid: "f19ede8bffa7471e9146e45d3fadd8ce" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 32865 } }
I20260925 07:30:51.604739 17538 ts_tablet_manager.cc:1428] T 8f24bd02d9234148a267c21f227d833c P 64749e2394c84a75819a9f88d38f903a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:51.605055 17557 raft_consensus.cc:2258] T d24fb2b06c8e4455a5e96a0b7a8c2a5e P 64749e2394c84a75819a9f88d38f903a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:30:51.605079 17534 tablet_bootstrap.cc:492] T 8f24bd02d9234148a267c21f227d833c P d68d1d5e4ad74b64b5e6b439a05d3fd7: No bootstrap required, opened a new log
I20260925 07:30:51.605118 17534 ts_tablet_manager.cc:1397] T 8f24bd02d9234148a267c21f227d833c P d68d1d5e4ad74b64b5e6b439a05d3fd7: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260925 07:30:51.605397 17534 raft_consensus.cc:348] T 8f24bd02d9234148a267c21f227d833c P d68d1d5e4ad74b64b5e6b439a05d3fd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d68d1d5e4ad74b64b5e6b439a05d3fd7" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 39919 } } peers { permanent_uuid: "64749e2394c84a75819a9f88d38f903a" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 33491 } } peers { permanent_uuid: "f19ede8bffa7471e9146e45d3fadd8ce" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 32865 } }
I20260925 07:30:51.605530 17534 raft_consensus.cc:374] T 8f24bd02d9234148a267c21f227d833c P d68d1d5e4ad74b64b5e6b439a05d3fd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:30:51.605559 17534 raft_consensus.cc:729] T 8f24bd02d9234148a267c21f227d833c P d68d1d5e4ad74b64b5e6b439a05d3fd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d68d1d5e4ad74b64b5e6b439a05d3fd7, State: Initialized, Role: FOLLOWER
I20260925 07:30:51.605605 17534 consensus_queue.cc:260] T 8f24bd02d9234148a267c21f227d833c P d68d1d5e4ad74b64b5e6b439a05d3fd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d68d1d5e4ad74b64b5e6b439a05d3fd7" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 39919 } } peers { permanent_uuid: "64749e2394c84a75819a9f88d38f903a" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 33491 } } peers { permanent_uuid: "f19ede8bffa7471e9146e45d3fadd8ce" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 32865 } }
I20260925 07:30:51.605731 17534 ts_tablet_manager.cc:1428] T 8f24bd02d9234148a267c21f227d833c P d68d1d5e4ad74b64b5e6b439a05d3fd7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 07:30:51.605756 17557 ts_tablet_manager.cc:1905] T d24fb2b06c8e4455a5e96a0b7a8c2a5e P 64749e2394c84a75819a9f88d38f903a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 07:30:51.607244 17557 ts_tablet_manager.cc:1918] T d24fb2b06c8e4455a5e96a0b7a8c2a5e P 64749e2394c84a75819a9f88d38f903a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 07:30:51.607316 17557 log.cc:1196] T d24fb2b06c8e4455a5e96a0b7a8c2a5e P 64749e2394c84a75819a9f88d38f903a: Deleting WAL directory at /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790321392278483-9480-0/minicluster-data/ts-1/wal/wals/d24fb2b06c8e4455a5e96a0b7a8c2a5e
I20260925 07:30:51.607579 17557 ts_tablet_manager.cc:1939] T d24fb2b06c8e4455a5e96a0b7a8c2a5e P 64749e2394c84a75819a9f88d38f903a: Deleting consensus metadata
I20260925 07:30:51.607929 17007 catalog_manager.cc:4841] TS 64749e2394c84a75819a9f88d38f903a (127.9.66.2:33491): tablet d24fb2b06c8e4455a5e96a0b7a8c2a5e (table test-table [id=810be3e3f7c643a9bf78b9e3243c3646]) successfully deleted
I20260925 07:30:51.691967 17541 raft_consensus.cc:482] T 8f24bd02d9234148a267c21f227d833c P 64749e2394c84a75819a9f88d38f903a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 07:30:51.692078 17541 raft_consensus.cc:504] T 8f24bd02d9234148a267c21f227d833c P 64749e2394c84a75819a9f88d38f903a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d68d1d5e4ad74b64b5e6b439a05d3fd7" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 39919 } } peers { permanent_uuid: "64749e2394c84a75819a9f88d38f903a" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 33491 } } peers { permanent_uuid: "f19ede8bffa7471e9146e45d3fadd8ce" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 32865 } }
I20260925 07:30:51.692406 17541 leader_election.cc:288] T 8f24bd02d9234148a267c21f227d833c P 64749e2394c84a75819a9f88d38f903a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d68d1d5e4ad74b64b5e6b439a05d3fd7 (127.9.66.1:39919), f19ede8bffa7471e9146e45d3fadd8ce (127.9.66.3:32865)
I20260925 07:30:51.695986 17528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:51.692884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.2:40923 (local address 127.9.66.3:32865)
0925 07:30:51.692933 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:51.692935 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:51.692943 (+ 8us) server_negotiation.cc:407] Connection header received
0925 07:30:51.692971 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:51.692972 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:51.692981 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:51.693057 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:30:51.693232 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:51.693963 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:51.695213 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:51.695571 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:51.695666 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:51.695676 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:51.695867 (+ 191us) server_negotiation.cc:299] Negotiation successful
0925 07:30:51.695909 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 07:30:51.695997 17562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:51.692769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.3:32865 (local address 127.9.66.2:40923)
0925 07:30:51.692851 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:51.692859 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 07:30:51.692898 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:30:51.693070 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:30:51.693071 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:30:51.693074 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:30:51.693145 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:51.693147 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:51.694046 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:51.694047 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:51.695091 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:51.695093 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:51.695671 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:51.695672 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:51.695750 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:51.695753 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 07:30:51.695849 (+ 96us) client_negotiation.cc:240] Negotiation successful
0925 07:30:51.695897 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71,"thread_start_us":41,"threads_started":1}
I20260925 07:30:51.696076 17530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:51.692697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.2:37145 (local address 127.9.66.1:39919)
0925 07:30:51.692768 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:51.692770 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:51.692846 (+ 76us) server_negotiation.cc:407] Connection header received
0925 07:30:51.692961 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:51.692962 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:51.692971 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:51.693006 (+ 35us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:30:51.693246 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:51.693788 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:51.695181 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:51.695717 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:51.695790 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:51.695799 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:51.695954 (+ 155us) server_negotiation.cc:299] Negotiation successful
0925 07:30:51.696000 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 07:30:51.696269 17561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:51.692599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.1:39919 (local address 127.9.66.2:37145)
0925 07:30:51.692774 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:51.692787 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 07:30:51.692878 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:30:51.693080 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:30:51.693081 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:30:51.693083 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:30:51.693143 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:51.693145 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:51.693869 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:51.693870 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:51.695084 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:51.695087 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:51.695801 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:51.695802 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:51.695869 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:51.695872 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 07:30:51.696175 (+ 303us) client_negotiation.cc:240] Negotiation successful
0925 07:30:51.696222 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"mutex_wait_us":16,"thread_start_us":68,"threads_started":1}
I20260925 07:30:51.696513 17179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f24bd02d9234148a267c21f227d833c" candidate_uuid: "64749e2394c84a75819a9f88d38f903a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d68d1d5e4ad74b64b5e6b439a05d3fd7" is_pre_election: true
I20260925 07:30:51.696611 17179 raft_consensus.cc:2454] T 8f24bd02d9234148a267c21f227d833c P d68d1d5e4ad74b64b5e6b439a05d3fd7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64749e2394c84a75819a9f88d38f903a in term 0.
I20260925 07:30:51.696583 17459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f24bd02d9234148a267c21f227d833c" candidate_uuid: "64749e2394c84a75819a9f88d38f903a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f19ede8bffa7471e9146e45d3fadd8ce" is_pre_election: true
I20260925 07:30:51.696686 17459 raft_consensus.cc:2454] T 8f24bd02d9234148a267c21f227d833c P f19ede8bffa7471e9146e45d3fadd8ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64749e2394c84a75819a9f88d38f903a in term 0.
I20260925 07:30:51.696863 17251 leader_election.cc:302] T 8f24bd02d9234148a267c21f227d833c P 64749e2394c84a75819a9f88d38f903a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64749e2394c84a75819a9f88d38f903a, d68d1d5e4ad74b64b5e6b439a05d3fd7; no voters:
I20260925 07:30:51.697019 17541 raft_consensus.cc:2781] T 8f24bd02d9234148a267c21f227d833c P 64749e2394c84a75819a9f88d38f903a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 07:30:51.697070 17541 raft_consensus.cc:482] T 8f24bd02d9234148a267c21f227d833c P 64749e2394c84a75819a9f88d38f903a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 07:30:51.697101 17541 raft_consensus.cc:3037] T 8f24bd02d9234148a267c21f227d833c P 64749e2394c84a75819a9f88d38f903a [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:30:51.697881 17541 raft_consensus.cc:504] T 8f24bd02d9234148a267c21f227d833c P 64749e2394c84a75819a9f88d38f903a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d68d1d5e4ad74b64b5e6b439a05d3fd7" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 39919 } } peers { permanent_uuid: "64749e2394c84a75819a9f88d38f903a" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 33491 } } peers { permanent_uuid: "f19ede8bffa7471e9146e45d3fadd8ce" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 32865 } }
I20260925 07:30:51.698010 17541 leader_election.cc:288] T 8f24bd02d9234148a267c21f227d833c P 64749e2394c84a75819a9f88d38f903a [CANDIDATE]: Term 1 election: Requested vote from peers d68d1d5e4ad74b64b5e6b439a05d3fd7 (127.9.66.1:39919), f19ede8bffa7471e9146e45d3fadd8ce (127.9.66.3:32865)
I20260925 07:30:51.698158 17179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f24bd02d9234148a267c21f227d833c" candidate_uuid: "64749e2394c84a75819a9f88d38f903a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d68d1d5e4ad74b64b5e6b439a05d3fd7"
I20260925 07:30:51.698199 17459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f24bd02d9234148a267c21f227d833c" candidate_uuid: "64749e2394c84a75819a9f88d38f903a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f19ede8bffa7471e9146e45d3fadd8ce"
I20260925 07:30:51.698261 17179 raft_consensus.cc:3037] T 8f24bd02d9234148a267c21f227d833c P d68d1d5e4ad74b64b5e6b439a05d3fd7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:30:51.698295 17459 raft_consensus.cc:3037] T 8f24bd02d9234148a267c21f227d833c P f19ede8bffa7471e9146e45d3fadd8ce [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:30:51.698889 17179 raft_consensus.cc:2454] T 8f24bd02d9234148a267c21f227d833c P d68d1d5e4ad74b64b5e6b439a05d3fd7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64749e2394c84a75819a9f88d38f903a in term 1.
I20260925 07:30:51.699023 17251 leader_election.cc:302] T 8f24bd02d9234148a267c21f227d833c P 64749e2394c84a75819a9f88d38f903a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64749e2394c84a75819a9f88d38f903a, d68d1d5e4ad74b64b5e6b439a05d3fd7; no voters:
I20260925 07:30:51.699090 17541 raft_consensus.cc:2781] T 8f24bd02d9234148a267c21f227d833c P 64749e2394c84a75819a9f88d38f903a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:30:51.699172 17459 raft_consensus.cc:2454] T 8f24bd02d9234148a267c21f227d833c P f19ede8bffa7471e9146e45d3fadd8ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64749e2394c84a75819a9f88d38f903a in term 1.
I20260925 07:30:51.699211 17541 raft_consensus.cc:686] T 8f24bd02d9234148a267c21f227d833c P 64749e2394c84a75819a9f88d38f903a [term 1 LEADER]: Becoming Leader. State: Replica: 64749e2394c84a75819a9f88d38f903a, State: Running, Role: LEADER
I20260925 07:30:51.699268 17541 consensus_queue.cc:237] T 8f24bd02d9234148a267c21f227d833c P 64749e2394c84a75819a9f88d38f903a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d68d1d5e4ad74b64b5e6b439a05d3fd7" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 39919 } } peers { permanent_uuid: "64749e2394c84a75819a9f88d38f903a" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 33491 } } peers { permanent_uuid: "f19ede8bffa7471e9146e45d3fadd8ce" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 32865 } }
I20260925 07:30:51.699903 17021 catalog_manager.cc:5495] T 8f24bd02d9234148a267c21f227d833c P 64749e2394c84a75819a9f88d38f903a reported cstate change: term changed from 0 to 1, leader changed from <none> to 64749e2394c84a75819a9f88d38f903a (127.9.66.2). New cstate: current_term: 1 leader_uuid: "64749e2394c84a75819a9f88d38f903a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d68d1d5e4ad74b64b5e6b439a05d3fd7" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 39919 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64749e2394c84a75819a9f88d38f903a" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 33491 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f19ede8bffa7471e9146e45d3fadd8ce" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 32865 } health_report { overall_health: UNKNOWN } } }
Sep 25 07:30:51 dist-test-slave-57fn krb5kdc[16977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321450, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.66.62@KRBTEST.COM for oryx/127.9.66.62@KRBTEST.COM
I20260925 07:30:51.740952 17570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:51.734173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34821 (local address 127.0.0.1:50622)
0925 07:30:51.734317 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:51.734328 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 07:30:51.734419 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:30:51.734564 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:30:51.734565 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:30:51.734887 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:30:51.734932 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:51.734935 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:51.735571 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:51.735572 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:51.736088 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:51.736089 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:51.736421 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:51.736422 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:51.736473 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:51.737044 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:30:51.737049 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:30:51.738516 (+ 1467us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:30:51.739949 (+ 1433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:30:51.739955 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:30:51.739957 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:30:51.740151 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:30:51.740340 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:30:51.740341 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:30:51.740341 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:30:51.740423 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:30:51.740652 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:30:51.740658 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:30:51.740705 (+ 47us) client_negotiation.cc:769] Sending connection context
0925 07:30:51.740773 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 07:30:51.740868 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":64,"threads_started":1}
I20260925 07:30:51.741040 17091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:51.734228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50622 (local address 127.9.66.62:34821)
0925 07:30:51.734277 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:51.734280 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:51.734402 (+ 122us) server_negotiation.cc:407] Connection header received
0925 07:30:51.734473 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:51.734474 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:51.734484 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:51.734559 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:30:51.735013 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:51.735497 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:51.736168 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:51.736343 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:51.736421 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:51.738614 (+ 2193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:30:51.738615 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:30:51.738615 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:30:51.738622 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:30:51.739868 (+ 1246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:51.740218 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:51.740218 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:51.740219 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:51.740265 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:51.740483 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:51.740484 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:51.740484 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:51.740581 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:30:51.740630 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:51.740798 (+ 168us) server_negotiation.cc:299] Negotiation successful
0925 07:30:51.740929 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":57,"server-negotiator.queue_time_us":19}
I20260925 07:30:51.742851 17021 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.9.66.62@KRBTEST.COM'} at 127.0.0.1:50622:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 07:30:51.743050 17021 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 07:30:51.744294 17159 tablet_service.cc:1467] Processing CreateTablet for tablet e5e6a48db5be4ef0a8b8a469d9c79bc5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=12c6787c52204155889604505ef6adfe]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 07:30:51.744323 17299 tablet_service.cc:1467] Processing CreateTablet for tablet e5e6a48db5be4ef0a8b8a469d9c79bc5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=12c6787c52204155889604505ef6adfe]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 07:30:51.744467 17159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5e6a48db5be4ef0a8b8a469d9c79bc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:30:51.744465 17299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5e6a48db5be4ef0a8b8a469d9c79bc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:30:51.745988 17538 tablet_bootstrap.cc:492] T e5e6a48db5be4ef0a8b8a469d9c79bc5 P 64749e2394c84a75819a9f88d38f903a: Bootstrap starting.
I20260925 07:30:51.746003 17534 tablet_bootstrap.cc:492] T e5e6a48db5be4ef0a8b8a469d9c79bc5 P d68d1d5e4ad74b64b5e6b439a05d3fd7: Bootstrap starting.
I20260925 07:30:51.746137 17439 tablet_service.cc:1467] Processing CreateTablet for tablet e5e6a48db5be4ef0a8b8a469d9c79bc5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=12c6787c52204155889604505ef6adfe]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 07:30:51.746266 17439 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5e6a48db5be4ef0a8b8a469d9c79bc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:30:51.746429 17534 tablet_bootstrap.cc:654] T e5e6a48db5be4ef0a8b8a469d9c79bc5 P d68d1d5e4ad74b64b5e6b439a05d3fd7: Neither blocks nor log segments found. Creating new log.
I20260925 07:30:51.746429 17538 tablet_bootstrap.cc:654] T e5e6a48db5be4ef0a8b8a469d9c79bc5 P 64749e2394c84a75819a9f88d38f903a: Neither blocks nor log segments found. Creating new log.
I20260925 07:30:51.747081 17538 tablet_bootstrap.cc:492] T e5e6a48db5be4ef0a8b8a469d9c79bc5 P 64749e2394c84a75819a9f88d38f903a: No bootstrap required, opened a new log
I20260925 07:30:51.747165 17538 ts_tablet_manager.cc:1397] T e5e6a48db5be4ef0a8b8a469d9c79bc5 P 64749e2394c84a75819a9f88d38f903a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 07:30:51.747269 17534 tablet_bootstrap.cc:492] T e5e6a48db5be4ef0a8b8a469d9c79bc5 P d68d1d5e4ad74b64b5e6b439a05d3fd7: No bootstrap required, opened a new log
I20260925 07:30:51.747304 17534 ts_tablet_manager.cc:1397] T e5e6a48db5be4ef0a8b8a469d9c79bc5 P d68d1d5e4ad74b64b5e6b439a05d3fd7: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 07:30:51.747318 17538 raft_consensus.cc:348] T e5e6a48db5be4ef0a8b8a469d9c79bc5 P 64749e2394c84a75819a9f88d38f903a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d68d1d5e4ad74b64b5e6b439a05d3fd7" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 39919 } } peers { permanent_uuid: "64749e2394c84a75819a9f88d38f903a" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 33491 } } peers { permanent_uuid: "f19ede8bffa7471e9146e45d3fadd8ce" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 32865 } }
I20260925 07:30:51.747391 17538 raft_consensus.cc:374] T e5e6a48db5be4ef0a8b8a469d9c79bc5 P 64749e2394c84a75819a9f88d38f903a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:30:51.747411 17538 raft_consensus.cc:729] T e5e6a48db5be4ef0a8b8a469d9c79bc5 P 64749e2394c84a75819a9f88d38f903a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64749e2394c84a75819a9f88d38f903a, State: Initialized, Role: FOLLOWER
I20260925 07:30:51.747430 17534 raft_consensus.cc:348] T e5e6a48db5be4ef0a8b8a469d9c79bc5 P d68d1d5e4ad74b64b5e6b439a05d3fd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d68d1d5e4ad74b64b5e6b439a05d3fd7" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 39919 } } peers { permanent_uuid: "64749e2394c84a75819a9f88d38f903a" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 33491 } } peers { permanent_uuid: "f19ede8bffa7471e9146e45d3fadd8ce" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 32865 } }
I20260925 07:30:51.747489 17534 raft_consensus.cc:374] T e5e6a48db5be4ef0a8b8a469d9c79bc5 P d68d1d5e4ad74b64b5e6b439a05d3fd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:30:51.747511 17534 raft_consensus.cc:729] T e5e6a48db5be4ef0a8b8a469d9c79bc5 P d68d1d5e4ad74b64b5e6b439a05d3fd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d68d1d5e4ad74b64b5e6b439a05d3fd7, State: Initialized, Role: FOLLOWER
I20260925 07:30:51.747478 17538 consensus_queue.cc:260] T e5e6a48db5be4ef0a8b8a469d9c79bc5 P 64749e2394c84a75819a9f88d38f903a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d68d1d5e4ad74b64b5e6b439a05d3fd7" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 39919 } } peers { permanent_uuid: "64749e2394c84a75819a9f88d38f903a" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 33491 } } peers { permanent_uuid: "f19ede8bffa7471e9146e45d3fadd8ce" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 32865 } }
I20260925 07:30:51.747555 17534 consensus_queue.cc:260] T e5e6a48db5be4ef0a8b8a469d9c79bc5 P d68d1d5e4ad74b64b5e6b439a05d3fd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d68d1d5e4ad74b64b5e6b439a05d3fd7" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 39919 } } peers { permanent_uuid: "64749e2394c84a75819a9f88d38f903a" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 33491 } } peers { permanent_uuid: "f19ede8bffa7471e9146e45d3fadd8ce" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 32865 } }
I20260925 07:30:51.747601 17538 ts_tablet_manager.cc:1428] T e5e6a48db5be4ef0a8b8a469d9c79bc5 P 64749e2394c84a75819a9f88d38f903a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:51.747637 17534 ts_tablet_manager.cc:1428] T e5e6a48db5be4ef0a8b8a469d9c79bc5 P d68d1d5e4ad74b64b5e6b439a05d3fd7: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:51.747684 17541 tablet_replica.cc:439] T e5e6a48db5be4ef0a8b8a469d9c79bc5 P 64749e2394c84a75819a9f88d38f903a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d68d1d5e4ad74b64b5e6b439a05d3fd7" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 39919 } } peers { permanent_uuid: "64749e2394c84a75819a9f88d38f903a" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 33491 } } peers { permanent_uuid: "f19ede8bffa7471e9146e45d3fadd8ce" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 32865 } } }
I20260925 07:30:51.747774 17541 tablet_replica.cc:442] T e5e6a48db5be4ef0a8b8a469d9c79bc5 P 64749e2394c84a75819a9f88d38f903a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 07:30:51.747865 17533 tablet_bootstrap.cc:492] T e5e6a48db5be4ef0a8b8a469d9c79bc5 P f19ede8bffa7471e9146e45d3fadd8ce: Bootstrap starting.
I20260925 07:30:51.748203 17533 tablet_bootstrap.cc:654] T e5e6a48db5be4ef0a8b8a469d9c79bc5 P f19ede8bffa7471e9146e45d3fadd8ce: Neither blocks nor log segments found. Creating new log.
I20260925 07:30:51.748454 17537 tablet_replica.cc:439] T e5e6a48db5be4ef0a8b8a469d9c79bc5 P d68d1d5e4ad74b64b5e6b439a05d3fd7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d68d1d5e4ad74b64b5e6b439a05d3fd7" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 39919 } } peers { permanent_uuid: "64749e2394c84a75819a9f88d38f903a" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 33491 } } peers { permanent_uuid: "f19ede8bffa7471e9146e45d3fadd8ce" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 32865 } } }
I20260925 07:30:51.748549 17537 tablet_replica.cc:442] T e5e6a48db5be4ef0a8b8a469d9c79bc5 P d68d1d5e4ad74b64b5e6b439a05d3fd7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 07:30:51.748730 17533 tablet_bootstrap.cc:492] T e5e6a48db5be4ef0a8b8a469d9c79bc5 P f19ede8bffa7471e9146e45d3fadd8ce: No bootstrap required, opened a new log
I20260925 07:30:51.748781 17533 ts_tablet_manager.cc:1397] T e5e6a48db5be4ef0a8b8a469d9c79bc5 P f19ede8bffa7471e9146e45d3fadd8ce: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 07:30:51.748919 17533 raft_consensus.cc:348] T e5e6a48db5be4ef0a8b8a469d9c79bc5 P f19ede8bffa7471e9146e45d3fadd8ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d68d1d5e4ad74b64b5e6b439a05d3fd7" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 39919 } } peers { permanent_uuid: "64749e2394c84a75819a9f88d38f903a" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 33491 } } peers { permanent_uuid: "f19ede8bffa7471e9146e45d3fadd8ce" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 32865 } }
I20260925 07:30:51.748980 17533 raft_consensus.cc:374] T e5e6a48db5be4ef0a8b8a469d9c79bc5 P f19ede8bffa7471e9146e45d3fadd8ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:30:51.748998 17533 raft_consensus.cc:729] T e5e6a48db5be4ef0a8b8a469d9c79bc5 P f19ede8bffa7471e9146e45d3fadd8ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f19ede8bffa7471e9146e45d3fadd8ce, State: Initialized, Role: FOLLOWER
I20260925 07:30:51.749039 17533 consensus_queue.cc:260] T e5e6a48db5be4ef0a8b8a469d9c79bc5 P f19ede8bffa7471e9146e45d3fadd8ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d68d1d5e4ad74b64b5e6b439a05d3fd7" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 39919 } } peers { permanent_uuid: "64749e2394c84a75819a9f88d38f903a" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 33491 } } peers { permanent_uuid: "f19ede8bffa7471e9146e45d3fadd8ce" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 32865 } }
I20260925 07:30:51.749101 17533 ts_tablet_manager.cc:1428] T e5e6a48db5be4ef0a8b8a469d9c79bc5 P f19ede8bffa7471e9146e45d3fadd8ce: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:51.749143 17543 tablet_replica.cc:439] T e5e6a48db5be4ef0a8b8a469d9c79bc5 P f19ede8bffa7471e9146e45d3fadd8ce: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d68d1d5e4ad74b64b5e6b439a05d3fd7" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 39919 } } peers { permanent_uuid: "64749e2394c84a75819a9f88d38f903a" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 33491 } } peers { permanent_uuid: "f19ede8bffa7471e9146e45d3fadd8ce" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 32865 } } }
I20260925 07:30:51.749243 17543 tablet_replica.cc:442] T e5e6a48db5be4ef0a8b8a469d9c79bc5 P f19ede8bffa7471e9146e45d3fadd8ce: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 07:30:51.976395 17541 raft_consensus.cc:482] T e5e6a48db5be4ef0a8b8a469d9c79bc5 P 64749e2394c84a75819a9f88d38f903a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 07:30:51.976480 17541 raft_consensus.cc:504] T e5e6a48db5be4ef0a8b8a469d9c79bc5 P 64749e2394c84a75819a9f88d38f903a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d68d1d5e4ad74b64b5e6b439a05d3fd7" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 39919 } } peers { permanent_uuid: "64749e2394c84a75819a9f88d38f903a" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 33491 } } peers { permanent_uuid: "f19ede8bffa7471e9146e45d3fadd8ce" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 32865 } }
I20260925 07:30:51.976627 17541 leader_election.cc:288] T e5e6a48db5be4ef0a8b8a469d9c79bc5 P 64749e2394c84a75819a9f88d38f903a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d68d1d5e4ad74b64b5e6b439a05d3fd7 (127.9.66.1:39919), f19ede8bffa7471e9146e45d3fadd8ce (127.9.66.3:32865)
I20260925 07:30:51.976902 17179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5e6a48db5be4ef0a8b8a469d9c79bc5" candidate_uuid: "64749e2394c84a75819a9f88d38f903a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d68d1d5e4ad74b64b5e6b439a05d3fd7" is_pre_election: true
I20260925 07:30:51.976981 17179 raft_consensus.cc:2454] T e5e6a48db5be4ef0a8b8a469d9c79bc5 P d68d1d5e4ad74b64b5e6b439a05d3fd7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64749e2394c84a75819a9f88d38f903a in term 0.
I20260925 07:30:51.976900 17459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5e6a48db5be4ef0a8b8a469d9c79bc5" candidate_uuid: "64749e2394c84a75819a9f88d38f903a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f19ede8bffa7471e9146e45d3fadd8ce" is_pre_election: true
I20260925 07:30:51.977030 17459 raft_consensus.cc:2454] T e5e6a48db5be4ef0a8b8a469d9c79bc5 P f19ede8bffa7471e9146e45d3fadd8ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64749e2394c84a75819a9f88d38f903a in term 0.
I20260925 07:30:51.977178 17251 leader_election.cc:302] T e5e6a48db5be4ef0a8b8a469d9c79bc5 P 64749e2394c84a75819a9f88d38f903a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64749e2394c84a75819a9f88d38f903a, d68d1d5e4ad74b64b5e6b439a05d3fd7; no voters:
I20260925 07:30:51.977270 17541 raft_consensus.cc:2781] T e5e6a48db5be4ef0a8b8a469d9c79bc5 P 64749e2394c84a75819a9f88d38f903a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 07:30:51.977310 17541 raft_consensus.cc:482] T e5e6a48db5be4ef0a8b8a469d9c79bc5 P 64749e2394c84a75819a9f88d38f903a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 07:30:51.977332 17541 raft_consensus.cc:3037] T e5e6a48db5be4ef0a8b8a469d9c79bc5 P 64749e2394c84a75819a9f88d38f903a [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:30:51.977977 17541 raft_consensus.cc:504] T e5e6a48db5be4ef0a8b8a469d9c79bc5 P 64749e2394c84a75819a9f88d38f903a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d68d1d5e4ad74b64b5e6b439a05d3fd7" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 39919 } } peers { permanent_uuid: "64749e2394c84a75819a9f88d38f903a" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 33491 } } peers { permanent_uuid: "f19ede8bffa7471e9146e45d3fadd8ce" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 32865 } }
I20260925 07:30:51.978096 17541 leader_election.cc:288] T e5e6a48db5be4ef0a8b8a469d9c79bc5 P 64749e2394c84a75819a9f88d38f903a [CANDIDATE]: Term 1 election: Requested vote from peers d68d1d5e4ad74b64b5e6b439a05d3fd7 (127.9.66.1:39919), f19ede8bffa7471e9146e45d3fadd8ce (127.9.66.3:32865)
I20260925 07:30:51.978250 17179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5e6a48db5be4ef0a8b8a469d9c79bc5" candidate_uuid: "64749e2394c84a75819a9f88d38f903a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d68d1d5e4ad74b64b5e6b439a05d3fd7"
I20260925 07:30:51.978271 17459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5e6a48db5be4ef0a8b8a469d9c79bc5" candidate_uuid: "64749e2394c84a75819a9f88d38f903a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f19ede8bffa7471e9146e45d3fadd8ce"
I20260925 07:30:51.978335 17459 raft_consensus.cc:3037] T e5e6a48db5be4ef0a8b8a469d9c79bc5 P f19ede8bffa7471e9146e45d3fadd8ce [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:30:51.978336 17179 raft_consensus.cc:3037] T e5e6a48db5be4ef0a8b8a469d9c79bc5 P d68d1d5e4ad74b64b5e6b439a05d3fd7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:30:51.979116 17179 raft_consensus.cc:2454] T e5e6a48db5be4ef0a8b8a469d9c79bc5 P d68d1d5e4ad74b64b5e6b439a05d3fd7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64749e2394c84a75819a9f88d38f903a in term 1.
I20260925 07:30:51.979143 17459 raft_consensus.cc:2454] T e5e6a48db5be4ef0a8b8a469d9c79bc5 P f19ede8bffa7471e9146e45d3fadd8ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64749e2394c84a75819a9f88d38f903a in term 1.
I20260925 07:30:51.979277 17251 leader_election.cc:302] T e5e6a48db5be4ef0a8b8a469d9c79bc5 P 64749e2394c84a75819a9f88d38f903a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64749e2394c84a75819a9f88d38f903a, d68d1d5e4ad74b64b5e6b439a05d3fd7; no voters:
I20260925 07:30:51.979393 17541 raft_consensus.cc:2781] T e5e6a48db5be4ef0a8b8a469d9c79bc5 P 64749e2394c84a75819a9f88d38f903a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:30:51.979462 17541 raft_consensus.cc:686] T e5e6a48db5be4ef0a8b8a469d9c79bc5 P 64749e2394c84a75819a9f88d38f903a [term 1 LEADER]: Becoming Leader. State: Replica: 64749e2394c84a75819a9f88d38f903a, State: Running, Role: LEADER
I20260925 07:30:51.979513 17541 consensus_queue.cc:237] T e5e6a48db5be4ef0a8b8a469d9c79bc5 P 64749e2394c84a75819a9f88d38f903a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d68d1d5e4ad74b64b5e6b439a05d3fd7" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 39919 } } peers { permanent_uuid: "64749e2394c84a75819a9f88d38f903a" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 33491 } } peers { permanent_uuid: "f19ede8bffa7471e9146e45d3fadd8ce" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 32865 } }
I20260925 07:30:51.979820 17563 tablet_replica.cc:439] T e5e6a48db5be4ef0a8b8a469d9c79bc5 P 64749e2394c84a75819a9f88d38f903a: TxnStatusTablet state changed. Reason: New leader 64749e2394c84a75819a9f88d38f903a. Latest consensus state: current_term: 1 leader_uuid: "64749e2394c84a75819a9f88d38f903a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d68d1d5e4ad74b64b5e6b439a05d3fd7" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 39919 } } peers { permanent_uuid: "64749e2394c84a75819a9f88d38f903a" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 33491 } } peers { permanent_uuid: "f19ede8bffa7471e9146e45d3fadd8ce" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 32865 } } }
I20260925 07:30:51.979921 17563 tablet_replica.cc:442] T e5e6a48db5be4ef0a8b8a469d9c79bc5 P 64749e2394c84a75819a9f88d38f903a: This TxnStatusTablet replica's current role is: LEADER
I20260925 07:30:51.980122 17575 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 07:30:51.980121 17021 catalog_manager.cc:5495] T e5e6a48db5be4ef0a8b8a469d9c79bc5 P 64749e2394c84a75819a9f88d38f903a reported cstate change: term changed from 0 to 1, leader changed from <none> to 64749e2394c84a75819a9f88d38f903a (127.9.66.2). New cstate: current_term: 1 leader_uuid: "64749e2394c84a75819a9f88d38f903a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d68d1d5e4ad74b64b5e6b439a05d3fd7" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 39919 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64749e2394c84a75819a9f88d38f903a" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 33491 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f19ede8bffa7471e9146e45d3fadd8ce" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 32865 } health_report { overall_health: UNKNOWN } } }
I20260925 07:30:51.980255 17575 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 07:30:52.030454 17570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:52.027898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.2:33491 (local address 127.0.0.1:55288)
0925 07:30:52.027967 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:52.027980 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 07:30:52.028032 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:30:52.028208 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:30:52.028209 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:30:52.028211 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 07:30:52.028260 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:52.028262 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:52.028918 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:52.028919 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:52.029487 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:52.029489 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:52.030004 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:52.030005 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:52.030051 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:52.030063 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 07:30:52.030327 (+ 264us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 07:30:52.030333 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 07:30:52.030360 (+ 27us) client_negotiation.cc:240] Negotiation successful
0925 07:30:52.030394 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260925 07:30:52.030471 17532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:52.027955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55288 (local address 127.9.66.2:33491)
0925 07:30:52.028014 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:52.028017 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:52.028027 (+ 10us) server_negotiation.cc:407] Connection header received
0925 07:30:52.028106 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:52.028107 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:52.028126 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:52.028202 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 07:30:52.028359 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:52.028836 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:52.029624 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:52.029933 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:52.030001 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:52.030155 (+ 154us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 07:30:52.030269 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 07:30:52.030335 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:52.030375 (+ 40us) server_negotiation.cc:299] Negotiation successful
0925 07:30:52.030395 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 07:30:52.031790 17179 raft_consensus.cc:1261] T e5e6a48db5be4ef0a8b8a469d9c79bc5 P d68d1d5e4ad74b64b5e6b439a05d3fd7 [term 1 FOLLOWER]: Refusing update from remote peer 64749e2394c84a75819a9f88d38f903a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:30:52.032063 17459 raft_consensus.cc:1261] T e5e6a48db5be4ef0a8b8a469d9c79bc5 P f19ede8bffa7471e9146e45d3fadd8ce [term 1 FOLLOWER]: Refusing update from remote peer 64749e2394c84a75819a9f88d38f903a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:30:52.032065 17541 consensus_queue.cc:1035] T e5e6a48db5be4ef0a8b8a469d9c79bc5 P 64749e2394c84a75819a9f88d38f903a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d68d1d5e4ad74b64b5e6b439a05d3fd7" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 39919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:30:52.032351 17541 consensus_queue.cc:1035] T e5e6a48db5be4ef0a8b8a469d9c79bc5 P 64749e2394c84a75819a9f88d38f903a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f19ede8bffa7471e9146e45d3fadd8ce" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 32865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:30:52.032624 17537 tablet_replica.cc:439] T e5e6a48db5be4ef0a8b8a469d9c79bc5 P d68d1d5e4ad74b64b5e6b439a05d3fd7: TxnStatusTablet state changed. Reason: New leader 64749e2394c84a75819a9f88d38f903a. Latest consensus state: current_term: 1 leader_uuid: "64749e2394c84a75819a9f88d38f903a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d68d1d5e4ad74b64b5e6b439a05d3fd7" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 39919 } } peers { permanent_uuid: "64749e2394c84a75819a9f88d38f903a" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 33491 } } peers { permanent_uuid: "f19ede8bffa7471e9146e45d3fadd8ce" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 32865 } } }
I20260925 07:30:52.032744 17537 tablet_replica.cc:442] T e5e6a48db5be4ef0a8b8a469d9c79bc5 P d68d1d5e4ad74b64b5e6b439a05d3fd7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 07:30:52.032927 17543 tablet_replica.cc:439] T e5e6a48db5be4ef0a8b8a469d9c79bc5 P f19ede8bffa7471e9146e45d3fadd8ce: TxnStatusTablet state changed. Reason: New leader 64749e2394c84a75819a9f88d38f903a. Latest consensus state: current_term: 1 leader_uuid: "64749e2394c84a75819a9f88d38f903a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d68d1d5e4ad74b64b5e6b439a05d3fd7" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 39919 } } peers { permanent_uuid: "64749e2394c84a75819a9f88d38f903a" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 33491 } } peers { permanent_uuid: "f19ede8bffa7471e9146e45d3fadd8ce" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 32865 } } }
I20260925 07:30:52.033006 17543 tablet_replica.cc:442] T e5e6a48db5be4ef0a8b8a469d9c79bc5 P f19ede8bffa7471e9146e45d3fadd8ce: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 07:30:52.033805 17563 tablet_replica.cc:439] T e5e6a48db5be4ef0a8b8a469d9c79bc5 P 64749e2394c84a75819a9f88d38f903a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "64749e2394c84a75819a9f88d38f903a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d68d1d5e4ad74b64b5e6b439a05d3fd7" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 39919 } } peers { permanent_uuid: "64749e2394c84a75819a9f88d38f903a" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 33491 } } peers { permanent_uuid: "f19ede8bffa7471e9146e45d3fadd8ce" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 32865 } } }
I20260925 07:30:52.033900 17563 tablet_replica.cc:442] T e5e6a48db5be4ef0a8b8a469d9c79bc5 P 64749e2394c84a75819a9f88d38f903a: This TxnStatusTablet replica's current role is: LEADER
I20260925 07:30:52.033871 17580 tablet_replica.cc:439] T e5e6a48db5be4ef0a8b8a469d9c79bc5 P 64749e2394c84a75819a9f88d38f903a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "64749e2394c84a75819a9f88d38f903a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d68d1d5e4ad74b64b5e6b439a05d3fd7" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 39919 } } peers { permanent_uuid: "64749e2394c84a75819a9f88d38f903a" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 33491 } } peers { permanent_uuid: "f19ede8bffa7471e9146e45d3fadd8ce" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 32865 } } }
I20260925 07:30:52.033949 17580 tablet_replica.cc:442] T e5e6a48db5be4ef0a8b8a469d9c79bc5 P 64749e2394c84a75819a9f88d38f903a: This TxnStatusTablet replica's current role is: LEADER
I20260925 07:30:52.034266 17546 mvcc.cc:205] Tried to move back new op lower bound from 7333156667520102400 to 7333156667308355584. Current Snapshot: MvccSnapshot[applied={T|T < 7333156667520102400}]
I20260925 07:30:52.034332 17537 tablet_replica.cc:439] T e5e6a48db5be4ef0a8b8a469d9c79bc5 P d68d1d5e4ad74b64b5e6b439a05d3fd7: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "64749e2394c84a75819a9f88d38f903a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d68d1d5e4ad74b64b5e6b439a05d3fd7" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 39919 } } peers { permanent_uuid: "64749e2394c84a75819a9f88d38f903a" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 33491 } } peers { permanent_uuid: "f19ede8bffa7471e9146e45d3fadd8ce" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 32865 } } }
I20260925 07:30:52.034408 17537 tablet_replica.cc:442] T e5e6a48db5be4ef0a8b8a469d9c79bc5 P d68d1d5e4ad74b64b5e6b439a05d3fd7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 07:30:52.034394 17543 tablet_replica.cc:439] T e5e6a48db5be4ef0a8b8a469d9c79bc5 P f19ede8bffa7471e9146e45d3fadd8ce: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "64749e2394c84a75819a9f88d38f903a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d68d1d5e4ad74b64b5e6b439a05d3fd7" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 39919 } } peers { permanent_uuid: "64749e2394c84a75819a9f88d38f903a" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 33491 } } peers { permanent_uuid: "f19ede8bffa7471e9146e45d3fadd8ce" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 32865 } } }
I20260925 07:30:52.034456 17543 tablet_replica.cc:442] T e5e6a48db5be4ef0a8b8a469d9c79bc5 P f19ede8bffa7471e9146e45d3fadd8ce: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 07:30:52.037824 17532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:52.035486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55292 (local address 127.9.66.2:33491)
0925 07:30:52.035522 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:52.035524 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:52.035532 (+ 8us) server_negotiation.cc:407] Connection header received
0925 07:30:52.035592 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:52.035593 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:52.035604 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:52.035673 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 07:30:52.035762 (+ 89us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:52.036445 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:52.037124 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:52.037392 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:52.037462 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:52.037603 (+ 141us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 07:30:52.037641 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 07:30:52.037692 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:52.037742 (+ 50us) server_negotiation.cc:299] Negotiation successful
0925 07:30:52.037762 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 07:30:52.042375 17532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:52.040023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.2:52113 (local address 127.9.66.2:33491)
0925 07:30:52.040056 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:52.040057 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:52.040195 (+ 138us) server_negotiation.cc:407] Connection header received
0925 07:30:52.040255 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:52.040256 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:52.040262 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:52.040305 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 07:30:52.040416 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:52.040822 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:52.041849 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:52.041998 (+ 149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:52.042044 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:52.042219 (+ 175us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 07:30:52.042238 (+ 19us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 07:30:52.042268 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:52.042323 (+ 55us) server_negotiation.cc:299] Negotiation successful
0925 07:30:52.042333 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260925 07:30:52.042459 17583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:52.039973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.2:33491 (local address 127.9.66.2:52113)
0925 07:30:52.040130 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:52.040140 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 07:30:52.040201 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:30:52.040310 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:30:52.040310 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:30:52.040312 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 07:30:52.040347 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:52.040349 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:52.040889 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:52.040890 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:52.041778 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:52.041780 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:52.042063 (+ 283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:52.042064 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:52.042114 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:52.042127 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 07:30:52.042272 (+ 145us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 07:30:52.042274 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 07:30:52.042322 (+ 48us) client_negotiation.cc:240] Negotiation successful
0925 07:30:52.042361 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":74,"threads_started":1}
I20260925 07:30:52.044404 17459 raft_consensus.cc:1261] T 8f24bd02d9234148a267c21f227d833c P f19ede8bffa7471e9146e45d3fadd8ce [term 1 FOLLOWER]: Refusing update from remote peer 64749e2394c84a75819a9f88d38f903a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:30:52.044430 17179 raft_consensus.cc:1261] T 8f24bd02d9234148a267c21f227d833c P d68d1d5e4ad74b64b5e6b439a05d3fd7 [term 1 FOLLOWER]: Refusing update from remote peer 64749e2394c84a75819a9f88d38f903a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:30:52.044569 17580 consensus_queue.cc:1035] T 8f24bd02d9234148a267c21f227d833c P 64749e2394c84a75819a9f88d38f903a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f19ede8bffa7471e9146e45d3fadd8ce" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 32865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:30:52.044668 17563 consensus_queue.cc:1035] T 8f24bd02d9234148a267c21f227d833c P 64749e2394c84a75819a9f88d38f903a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d68d1d5e4ad74b64b5e6b439a05d3fd7" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 39919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:30:52.056109 17020 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50606:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 07:30:52.056209 17020 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50606:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 07:30:52.056854 17020 catalog_manager.cc:5782] T 00000000000000000000000000000000 P df6bdd3c1abe434eb57ce7f7b79dd302: Sending DeleteTablet for 3 replicas of tablet 8f24bd02d9234148a267c21f227d833c
I20260925 07:30:52.057072 17159 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f24bd02d9234148a267c21f227d833c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:30:52 UTC) from {username='oryx', principal='oryx/127.9.66.62@KRBTEST.COM'} at 127.0.0.1:50720
I20260925 07:30:52.057129 17439 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f24bd02d9234148a267c21f227d833c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:30:52 UTC) from {username='oryx', principal='oryx/127.9.66.62@KRBTEST.COM'} at 127.0.0.1:35462
I20260925 07:30:52.057165 17299 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f24bd02d9234148a267c21f227d833c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:30:52 UTC) from {username='oryx', principal='oryx/127.9.66.62@KRBTEST.COM'} at 127.0.0.1:55282
I20260925 07:30:52.057183 17556 tablet_replica.cc:330] T 8f24bd02d9234148a267c21f227d833c P d68d1d5e4ad74b64b5e6b439a05d3fd7: stopping tablet replica
I20260925 07:30:52.057236 17556 raft_consensus.cc:2229] T 8f24bd02d9234148a267c21f227d833c P d68d1d5e4ad74b64b5e6b439a05d3fd7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 07:30:52.057245 17557 tablet_replica.cc:330] T 8f24bd02d9234148a267c21f227d833c P 64749e2394c84a75819a9f88d38f903a: stopping tablet replica
I20260925 07:30:52.057272 17556 raft_consensus.cc:2258] T 8f24bd02d9234148a267c21f227d833c P d68d1d5e4ad74b64b5e6b439a05d3fd7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:30:52.057271 17555 tablet_replica.cc:330] T 8f24bd02d9234148a267c21f227d833c P f19ede8bffa7471e9146e45d3fadd8ce: stopping tablet replica
I20260925 07:30:52.057287 17557 raft_consensus.cc:2229] T 8f24bd02d9234148a267c21f227d833c P 64749e2394c84a75819a9f88d38f903a [term 1 LEADER]: Raft consensus shutting down.
I20260925 07:30:52.057317 17555 raft_consensus.cc:2229] T 8f24bd02d9234148a267c21f227d833c P f19ede8bffa7471e9146e45d3fadd8ce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 07:30:52.057350 17555 raft_consensus.cc:2258] T 8f24bd02d9234148a267c21f227d833c P f19ede8bffa7471e9146e45d3fadd8ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:30:52.057358 17557 raft_consensus.cc:2258] T 8f24bd02d9234148a267c21f227d833c P 64749e2394c84a75819a9f88d38f903a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:30:52.057649 17557 ts_tablet_manager.cc:1905] T 8f24bd02d9234148a267c21f227d833c P 64749e2394c84a75819a9f88d38f903a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 07:30:52.057788 17555 ts_tablet_manager.cc:1905] T 8f24bd02d9234148a267c21f227d833c P f19ede8bffa7471e9146e45d3fadd8ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 07:30:52.059414 17557 ts_tablet_manager.cc:1918] T 8f24bd02d9234148a267c21f227d833c P 64749e2394c84a75819a9f88d38f903a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 07:30:52.059427 17555 ts_tablet_manager.cc:1918] T 8f24bd02d9234148a267c21f227d833c P f19ede8bffa7471e9146e45d3fadd8ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 07:30:52.059473 17555 log.cc:1196] T 8f24bd02d9234148a267c21f227d833c P f19ede8bffa7471e9146e45d3fadd8ce: Deleting WAL directory at /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790321392278483-9480-0/minicluster-data/ts-2/wal/wals/8f24bd02d9234148a267c21f227d833c
I20260925 07:30:52.059499 17557 log.cc:1196] T 8f24bd02d9234148a267c21f227d833c P 64749e2394c84a75819a9f88d38f903a: Deleting WAL directory at /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790321392278483-9480-0/minicluster-data/ts-1/wal/wals/8f24bd02d9234148a267c21f227d833c
I20260925 07:30:52.059700 17555 ts_tablet_manager.cc:1939] T 8f24bd02d9234148a267c21f227d833c P f19ede8bffa7471e9146e45d3fadd8ce: Deleting consensus metadata
I20260925 07:30:52.059706 17557 ts_tablet_manager.cc:1939] T 8f24bd02d9234148a267c21f227d833c P 64749e2394c84a75819a9f88d38f903a: Deleting consensus metadata
I20260925 07:30:52.059969 17007 catalog_manager.cc:4841] TS 64749e2394c84a75819a9f88d38f903a (127.9.66.2:33491): tablet 8f24bd02d9234148a267c21f227d833c (table test-table [id=5c67774f8c1e494182a71992d4375c29]) successfully deleted
I20260925 07:30:52.060046 17007 catalog_manager.cc:4841] TS f19ede8bffa7471e9146e45d3fadd8ce (127.9.66.3:32865): tablet 8f24bd02d9234148a267c21f227d833c (table test-table [id=5c67774f8c1e494182a71992d4375c29]) successfully deleted
I20260925 07:30:52.062222 17556 ts_tablet_manager.cc:1905] T 8f24bd02d9234148a267c21f227d833c P d68d1d5e4ad74b64b5e6b439a05d3fd7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 07:30:52.063795 17556 ts_tablet_manager.cc:1918] T 8f24bd02d9234148a267c21f227d833c P d68d1d5e4ad74b64b5e6b439a05d3fd7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 07:30:52.063860 17556 log.cc:1196] T 8f24bd02d9234148a267c21f227d833c P d68d1d5e4ad74b64b5e6b439a05d3fd7: Deleting WAL directory at /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790321392278483-9480-0/minicluster-data/ts-0/wal/wals/8f24bd02d9234148a267c21f227d833c
I20260925 07:30:52.064070 17556 ts_tablet_manager.cc:1939] T 8f24bd02d9234148a267c21f227d833c P d68d1d5e4ad74b64b5e6b439a05d3fd7: Deleting consensus metadata
I20260925 07:30:52.064368 17007 catalog_manager.cc:4841] TS d68d1d5e4ad74b64b5e6b439a05d3fd7 (127.9.66.1:39919): tablet 8f24bd02d9234148a267c21f227d833c (table test-table [id=5c67774f8c1e494182a71992d4375c29]) successfully deleted
Sep 25 07:30:52 dist-test-slave-57fn krb5kdc[16977](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321452, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 07:30:52 dist-test-slave-57fn krb5kdc[16977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.9.66.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 07:30:52 dist-test-slave-57fn krb5kdc[16977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.9.66.62@KRBTEST.COM, Server not found in Kerberos database
W20260925 07:30:52.085318 17091 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:30:52.079146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50626 (local address 127.9.66.62:34821)
0925 07:30:52.079204 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:52.079206 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:52.079310 (+ 104us) server_negotiation.cc:407] Connection header received
0925 07:30:52.079376 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:52.079377 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:52.079386 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:52.079437 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:30:52.080122 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:52.080584 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:52.081753 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:52.081974 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:52.082027 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:52.085235 (+ 3208us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50626: BlockingRecv error: recv got EOF from 127.0.0.1:50626 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 25 07:30:52 dist-test-slave-57fn krb5kdc[16977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321452, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.66.62@KRBTEST.COM
I20260925 07:30:52.093588 17091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:52.086290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50642 (local address 127.9.66.62:34821)
0925 07:30:52.086335 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:52.086337 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:52.086401 (+ 64us) server_negotiation.cc:407] Connection header received
0925 07:30:52.086464 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:52.086465 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:52.086472 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:52.086513 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:30:52.087057 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:52.087559 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:52.088557 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:52.088747 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:52.088787 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:52.091045 (+ 2258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:30:52.091046 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:30:52.091047 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:30:52.091054 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:30:52.092449 (+ 1395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:52.092798 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:52.092800 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:52.092800 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:52.092858 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:52.093055 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:52.093056 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:52.093057 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:52.093164 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:30:52.093261 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:52.093369 (+ 108us) server_negotiation.cc:299] Negotiation successful
0925 07:30:52.093449 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 07:30:52.095208 17020 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50642:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 07:30:52.095391 17020 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 07:30:52.096961 17439 tablet_service.cc:1467] Processing CreateTablet for tablet a81c27cf5fc6482e8f92b8104f22d7fc (DEFAULT_TABLE table=test-table [id=c50023367a4a4fbe869fd4c41faa200e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:30:52.096971 17299 tablet_service.cc:1467] Processing CreateTablet for tablet a81c27cf5fc6482e8f92b8104f22d7fc (DEFAULT_TABLE table=test-table [id=c50023367a4a4fbe869fd4c41faa200e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:30:52.096968 17159 tablet_service.cc:1467] Processing CreateTablet for tablet a81c27cf5fc6482e8f92b8104f22d7fc (DEFAULT_TABLE table=test-table [id=c50023367a4a4fbe869fd4c41faa200e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:30:52.097097 17299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a81c27cf5fc6482e8f92b8104f22d7fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:30:52.097097 17159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a81c27cf5fc6482e8f92b8104f22d7fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:30:52.097098 17439 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a81c27cf5fc6482e8f92b8104f22d7fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:30:52.098479 17534 tablet_bootstrap.cc:492] T a81c27cf5fc6482e8f92b8104f22d7fc P d68d1d5e4ad74b64b5e6b439a05d3fd7: Bootstrap starting.
I20260925 07:30:52.098493 17538 tablet_bootstrap.cc:492] T a81c27cf5fc6482e8f92b8104f22d7fc P 64749e2394c84a75819a9f88d38f903a: Bootstrap starting.
I20260925 07:30:52.098892 17534 tablet_bootstrap.cc:654] T a81c27cf5fc6482e8f92b8104f22d7fc P d68d1d5e4ad74b64b5e6b439a05d3fd7: Neither blocks nor log segments found. Creating new log.
I20260925 07:30:52.098922 17538 tablet_bootstrap.cc:654] T a81c27cf5fc6482e8f92b8104f22d7fc P 64749e2394c84a75819a9f88d38f903a: Neither blocks nor log segments found. Creating new log.
I20260925 07:30:52.099498 17533 tablet_bootstrap.cc:492] T a81c27cf5fc6482e8f92b8104f22d7fc P f19ede8bffa7471e9146e45d3fadd8ce: Bootstrap starting.
I20260925 07:30:52.099599 17534 tablet_bootstrap.cc:492] T a81c27cf5fc6482e8f92b8104f22d7fc P d68d1d5e4ad74b64b5e6b439a05d3fd7: No bootstrap required, opened a new log
I20260925 07:30:52.099642 17534 ts_tablet_manager.cc:1397] T a81c27cf5fc6482e8f92b8104f22d7fc P d68d1d5e4ad74b64b5e6b439a05d3fd7: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 07:30:52.099800 17534 raft_consensus.cc:348] T a81c27cf5fc6482e8f92b8104f22d7fc P d68d1d5e4ad74b64b5e6b439a05d3fd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d68d1d5e4ad74b64b5e6b439a05d3fd7" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 39919 } } peers { permanent_uuid: "64749e2394c84a75819a9f88d38f903a" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 33491 } } peers { permanent_uuid: "f19ede8bffa7471e9146e45d3fadd8ce" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 32865 } }
I20260925 07:30:52.099888 17534 raft_consensus.cc:374] T a81c27cf5fc6482e8f92b8104f22d7fc P d68d1d5e4ad74b64b5e6b439a05d3fd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:30:52.099910 17534 raft_consensus.cc:729] T a81c27cf5fc6482e8f92b8104f22d7fc P d68d1d5e4ad74b64b5e6b439a05d3fd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d68d1d5e4ad74b64b5e6b439a05d3fd7, State: Initialized, Role: FOLLOWER
I20260925 07:30:52.099905 17533 tablet_bootstrap.cc:654] T a81c27cf5fc6482e8f92b8104f22d7fc P f19ede8bffa7471e9146e45d3fadd8ce: Neither blocks nor log segments found. Creating new log.
I20260925 07:30:52.099960 17534 consensus_queue.cc:260] T a81c27cf5fc6482e8f92b8104f22d7fc P d68d1d5e4ad74b64b5e6b439a05d3fd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d68d1d5e4ad74b64b5e6b439a05d3fd7" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 39919 } } peers { permanent_uuid: "64749e2394c84a75819a9f88d38f903a" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 33491 } } peers { permanent_uuid: "f19ede8bffa7471e9146e45d3fadd8ce" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 32865 } }
I20260925 07:30:52.100090 17534 ts_tablet_manager.cc:1428] T a81c27cf5fc6482e8f92b8104f22d7fc P d68d1d5e4ad74b64b5e6b439a05d3fd7: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:52.100575 17533 tablet_bootstrap.cc:492] T a81c27cf5fc6482e8f92b8104f22d7fc P f19ede8bffa7471e9146e45d3fadd8ce: No bootstrap required, opened a new log
I20260925 07:30:52.100628 17533 ts_tablet_manager.cc:1397] T a81c27cf5fc6482e8f92b8104f22d7fc P f19ede8bffa7471e9146e45d3fadd8ce: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 07:30:52.100741 17533 raft_consensus.cc:348] T a81c27cf5fc6482e8f92b8104f22d7fc P f19ede8bffa7471e9146e45d3fadd8ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d68d1d5e4ad74b64b5e6b439a05d3fd7" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 39919 } } peers { permanent_uuid: "64749e2394c84a75819a9f88d38f903a" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 33491 } } peers { permanent_uuid: "f19ede8bffa7471e9146e45d3fadd8ce" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 32865 } }
I20260925 07:30:52.100805 17533 raft_consensus.cc:374] T a81c27cf5fc6482e8f92b8104f22d7fc P f19ede8bffa7471e9146e45d3fadd8ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:30:52.100829 17533 raft_consensus.cc:729] T a81c27cf5fc6482e8f92b8104f22d7fc P f19ede8bffa7471e9146e45d3fadd8ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f19ede8bffa7471e9146e45d3fadd8ce, State: Initialized, Role: FOLLOWER
I20260925 07:30:52.100883 17533 consensus_queue.cc:260] T a81c27cf5fc6482e8f92b8104f22d7fc P f19ede8bffa7471e9146e45d3fadd8ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d68d1d5e4ad74b64b5e6b439a05d3fd7" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 39919 } } peers { permanent_uuid: "64749e2394c84a75819a9f88d38f903a" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 33491 } } peers { permanent_uuid: "f19ede8bffa7471e9146e45d3fadd8ce" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 32865 } }
I20260925 07:30:52.100983 17533 ts_tablet_manager.cc:1428] T a81c27cf5fc6482e8f92b8104f22d7fc P f19ede8bffa7471e9146e45d3fadd8ce: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:52.101722 17538 tablet_bootstrap.cc:492] T a81c27cf5fc6482e8f92b8104f22d7fc P 64749e2394c84a75819a9f88d38f903a: No bootstrap required, opened a new log
I20260925 07:30:52.101781 17538 ts_tablet_manager.cc:1397] T a81c27cf5fc6482e8f92b8104f22d7fc P 64749e2394c84a75819a9f88d38f903a: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260925 07:30:52.101943 17538 raft_consensus.cc:348] T a81c27cf5fc6482e8f92b8104f22d7fc P 64749e2394c84a75819a9f88d38f903a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d68d1d5e4ad74b64b5e6b439a05d3fd7" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 39919 } } peers { permanent_uuid: "64749e2394c84a75819a9f88d38f903a" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 33491 } } peers { permanent_uuid: "f19ede8bffa7471e9146e45d3fadd8ce" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 32865 } }
I20260925 07:30:52.102020 17538 raft_consensus.cc:374] T a81c27cf5fc6482e8f92b8104f22d7fc P 64749e2394c84a75819a9f88d38f903a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:30:52.102043 17538 raft_consensus.cc:729] T a81c27cf5fc6482e8f92b8104f22d7fc P 64749e2394c84a75819a9f88d38f903a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64749e2394c84a75819a9f88d38f903a, State: Initialized, Role: FOLLOWER
I20260925 07:30:52.102092 17538 consensus_queue.cc:260] T a81c27cf5fc6482e8f92b8104f22d7fc P 64749e2394c84a75819a9f88d38f903a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d68d1d5e4ad74b64b5e6b439a05d3fd7" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 39919 } } peers { permanent_uuid: "64749e2394c84a75819a9f88d38f903a" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 33491 } } peers { permanent_uuid: "f19ede8bffa7471e9146e45d3fadd8ce" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 32865 } }
I20260925 07:30:52.102186 17538 ts_tablet_manager.cc:1428] T a81c27cf5fc6482e8f92b8104f22d7fc P 64749e2394c84a75819a9f88d38f903a: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 07:30:52.258407 17543 raft_consensus.cc:482] T a81c27cf5fc6482e8f92b8104f22d7fc P f19ede8bffa7471e9146e45d3fadd8ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 07:30:52.258507 17543 raft_consensus.cc:504] T a81c27cf5fc6482e8f92b8104f22d7fc P f19ede8bffa7471e9146e45d3fadd8ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d68d1d5e4ad74b64b5e6b439a05d3fd7" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 39919 } } peers { permanent_uuid: "64749e2394c84a75819a9f88d38f903a" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 33491 } } peers { permanent_uuid: "f19ede8bffa7471e9146e45d3fadd8ce" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 32865 } }
I20260925 07:30:52.258658 17543 leader_election.cc:288] T a81c27cf5fc6482e8f92b8104f22d7fc P f19ede8bffa7471e9146e45d3fadd8ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d68d1d5e4ad74b64b5e6b439a05d3fd7 (127.9.66.1:39919), 64749e2394c84a75819a9f88d38f903a (127.9.66.2:33491)
I20260925 07:30:52.258921 17179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a81c27cf5fc6482e8f92b8104f22d7fc" candidate_uuid: "f19ede8bffa7471e9146e45d3fadd8ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d68d1d5e4ad74b64b5e6b439a05d3fd7" is_pre_election: true
I20260925 07:30:52.259001 17179 raft_consensus.cc:2454] T a81c27cf5fc6482e8f92b8104f22d7fc P d68d1d5e4ad74b64b5e6b439a05d3fd7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f19ede8bffa7471e9146e45d3fadd8ce in term 0.
I20260925 07:30:52.258947 17319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a81c27cf5fc6482e8f92b8104f22d7fc" candidate_uuid: "f19ede8bffa7471e9146e45d3fadd8ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64749e2394c84a75819a9f88d38f903a" is_pre_election: true
I20260925 07:30:52.259107 17319 raft_consensus.cc:2454] T a81c27cf5fc6482e8f92b8104f22d7fc P 64749e2394c84a75819a9f88d38f903a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f19ede8bffa7471e9146e45d3fadd8ce in term 0.
I20260925 07:30:52.259191 17391 leader_election.cc:302] T a81c27cf5fc6482e8f92b8104f22d7fc P f19ede8bffa7471e9146e45d3fadd8ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d68d1d5e4ad74b64b5e6b439a05d3fd7, f19ede8bffa7471e9146e45d3fadd8ce; no voters:
I20260925 07:30:52.259279 17543 raft_consensus.cc:2781] T a81c27cf5fc6482e8f92b8104f22d7fc P f19ede8bffa7471e9146e45d3fadd8ce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 07:30:52.259316 17543 raft_consensus.cc:482] T a81c27cf5fc6482e8f92b8104f22d7fc P f19ede8bffa7471e9146e45d3fadd8ce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 07:30:52.259342 17543 raft_consensus.cc:3037] T a81c27cf5fc6482e8f92b8104f22d7fc P f19ede8bffa7471e9146e45d3fadd8ce [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:30:52.259999 17543 raft_consensus.cc:504] T a81c27cf5fc6482e8f92b8104f22d7fc P f19ede8bffa7471e9146e45d3fadd8ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d68d1d5e4ad74b64b5e6b439a05d3fd7" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 39919 } } peers { permanent_uuid: "64749e2394c84a75819a9f88d38f903a" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 33491 } } peers { permanent_uuid: "f19ede8bffa7471e9146e45d3fadd8ce" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 32865 } }
I20260925 07:30:52.260125 17543 leader_election.cc:288] T a81c27cf5fc6482e8f92b8104f22d7fc P f19ede8bffa7471e9146e45d3fadd8ce [CANDIDATE]: Term 1 election: Requested vote from peers d68d1d5e4ad74b64b5e6b439a05d3fd7 (127.9.66.1:39919), 64749e2394c84a75819a9f88d38f903a (127.9.66.2:33491)
I20260925 07:30:52.260277 17179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a81c27cf5fc6482e8f92b8104f22d7fc" candidate_uuid: "f19ede8bffa7471e9146e45d3fadd8ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d68d1d5e4ad74b64b5e6b439a05d3fd7"
I20260925 07:30:52.260286 17319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a81c27cf5fc6482e8f92b8104f22d7fc" candidate_uuid: "f19ede8bffa7471e9146e45d3fadd8ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64749e2394c84a75819a9f88d38f903a"
I20260925 07:30:52.260347 17179 raft_consensus.cc:3037] T a81c27cf5fc6482e8f92b8104f22d7fc P d68d1d5e4ad74b64b5e6b439a05d3fd7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:30:52.260372 17319 raft_consensus.cc:3037] T a81c27cf5fc6482e8f92b8104f22d7fc P 64749e2394c84a75819a9f88d38f903a [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:30:52.260922 17179 raft_consensus.cc:2454] T a81c27cf5fc6482e8f92b8104f22d7fc P d68d1d5e4ad74b64b5e6b439a05d3fd7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f19ede8bffa7471e9146e45d3fadd8ce in term 1.
I20260925 07:30:52.261092 17391 leader_election.cc:302] T a81c27cf5fc6482e8f92b8104f22d7fc P f19ede8bffa7471e9146e45d3fadd8ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d68d1d5e4ad74b64b5e6b439a05d3fd7, f19ede8bffa7471e9146e45d3fadd8ce; no voters:
I20260925 07:30:52.261179 17543 raft_consensus.cc:2781] T a81c27cf5fc6482e8f92b8104f22d7fc P f19ede8bffa7471e9146e45d3fadd8ce [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:30:52.261160 17319 raft_consensus.cc:2454] T a81c27cf5fc6482e8f92b8104f22d7fc P 64749e2394c84a75819a9f88d38f903a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f19ede8bffa7471e9146e45d3fadd8ce in term 1.
I20260925 07:30:52.261301 17543 raft_consensus.cc:686] T a81c27cf5fc6482e8f92b8104f22d7fc P f19ede8bffa7471e9146e45d3fadd8ce [term 1 LEADER]: Becoming Leader. State: Replica: f19ede8bffa7471e9146e45d3fadd8ce, State: Running, Role: LEADER
I20260925 07:30:52.261354 17543 consensus_queue.cc:237] T a81c27cf5fc6482e8f92b8104f22d7fc P f19ede8bffa7471e9146e45d3fadd8ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d68d1d5e4ad74b64b5e6b439a05d3fd7" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 39919 } } peers { permanent_uuid: "64749e2394c84a75819a9f88d38f903a" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 33491 } } peers { permanent_uuid: "f19ede8bffa7471e9146e45d3fadd8ce" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 32865 } }
I20260925 07:30:52.261862 17021 catalog_manager.cc:5495] T a81c27cf5fc6482e8f92b8104f22d7fc P f19ede8bffa7471e9146e45d3fadd8ce reported cstate change: term changed from 0 to 1, leader changed from <none> to f19ede8bffa7471e9146e45d3fadd8ce (127.9.66.3). New cstate: current_term: 1 leader_uuid: "f19ede8bffa7471e9146e45d3fadd8ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d68d1d5e4ad74b64b5e6b439a05d3fd7" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 39919 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64749e2394c84a75819a9f88d38f903a" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 33491 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f19ede8bffa7471e9146e45d3fadd8ce" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 32865 } health_report { overall_health: HEALTHY } } }
I20260925 07:30:52.283425 17605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:52.281233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35476 (local address 127.9.66.3:32865)
0925 07:30:52.281353 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:52.281354 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:52.281361 (+ 7us) server_negotiation.cc:407] Connection header received
0925 07:30:52.281381 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:52.281382 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:52.281391 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:52.281447 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 07:30:52.281608 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:52.282129 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:52.282910 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:52.283105 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:52.283157 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:52.283266 (+ 109us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 07:30:52.283289 (+ 23us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 07:30:52.283321 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:52.283350 (+ 29us) server_negotiation.cc:299] Negotiation successful
0925 07:30:52.283364 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":78,"threads_started":1}
I20260925 07:30:52.284173 17319 raft_consensus.cc:1261] T a81c27cf5fc6482e8f92b8104f22d7fc P 64749e2394c84a75819a9f88d38f903a [term 1 FOLLOWER]: Refusing update from remote peer f19ede8bffa7471e9146e45d3fadd8ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:30:52.284170 17179 raft_consensus.cc:1261] T a81c27cf5fc6482e8f92b8104f22d7fc P d68d1d5e4ad74b64b5e6b439a05d3fd7 [term 1 FOLLOWER]: Refusing update from remote peer f19ede8bffa7471e9146e45d3fadd8ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:30:52.284389 17543 consensus_queue.cc:1035] T a81c27cf5fc6482e8f92b8104f22d7fc P f19ede8bffa7471e9146e45d3fadd8ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "64749e2394c84a75819a9f88d38f903a" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 33491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:30:52.284523 17602 consensus_queue.cc:1035] T a81c27cf5fc6482e8f92b8104f22d7fc P f19ede8bffa7471e9146e45d3fadd8ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "d68d1d5e4ad74b64b5e6b439a05d3fd7" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 39919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:30:52.286932 17020 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50642:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 07:30:52.287016 17020 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50642:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 07:30:52.287585 17020 catalog_manager.cc:5782] T 00000000000000000000000000000000 P df6bdd3c1abe434eb57ce7f7b79dd302: Sending DeleteTablet for 3 replicas of tablet a81c27cf5fc6482e8f92b8104f22d7fc
I20260925 07:30:52.287819 17159 tablet_service.cc:1514] Processing DeleteTablet for tablet a81c27cf5fc6482e8f92b8104f22d7fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:30:52 UTC) from {username='oryx', principal='oryx/127.9.66.62@KRBTEST.COM'} at 127.0.0.1:50720
I20260925 07:30:52.287849 17439 tablet_service.cc:1514] Processing DeleteTablet for tablet a81c27cf5fc6482e8f92b8104f22d7fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:30:52 UTC) from {username='oryx', principal='oryx/127.9.66.62@KRBTEST.COM'} at 127.0.0.1:35462
I20260925 07:30:52.287916 17299 tablet_service.cc:1514] Processing DeleteTablet for tablet a81c27cf5fc6482e8f92b8104f22d7fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:30:52 UTC) from {username='oryx', principal='oryx/127.9.66.62@KRBTEST.COM'} at 127.0.0.1:55282
I20260925 07:30:52.288000 17556 tablet_replica.cc:330] T a81c27cf5fc6482e8f92b8104f22d7fc P d68d1d5e4ad74b64b5e6b439a05d3fd7: stopping tablet replica
I20260925 07:30:52.288057 17556 raft_consensus.cc:2229] T a81c27cf5fc6482e8f92b8104f22d7fc P d68d1d5e4ad74b64b5e6b439a05d3fd7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 07:30:52.288101 17556 raft_consensus.cc:2258] T a81c27cf5fc6482e8f92b8104f22d7fc P d68d1d5e4ad74b64b5e6b439a05d3fd7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:30:52.288177 17555 tablet_replica.cc:330] T a81c27cf5fc6482e8f92b8104f22d7fc P f19ede8bffa7471e9146e45d3fadd8ce: stopping tablet replica
I20260925 07:30:52.288126 17020 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50642:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 07:30:52.288242 17555 raft_consensus.cc:2229] T a81c27cf5fc6482e8f92b8104f22d7fc P f19ede8bffa7471e9146e45d3fadd8ce [term 1 LEADER]: Raft consensus shutting down.
W20260925 07:30:52.288265 17020 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 07:30:52.288290 17555 raft_consensus.cc:2258] T a81c27cf5fc6482e8f92b8104f22d7fc P f19ede8bffa7471e9146e45d3fadd8ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:30:52.288367 17557 tablet_replica.cc:330] T a81c27cf5fc6482e8f92b8104f22d7fc P 64749e2394c84a75819a9f88d38f903a: stopping tablet replica
I20260925 07:30:52.288427 17557 raft_consensus.cc:2229] T a81c27cf5fc6482e8f92b8104f22d7fc P 64749e2394c84a75819a9f88d38f903a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 07:30:52.288492 17557 raft_consensus.cc:2258] T a81c27cf5fc6482e8f92b8104f22d7fc P 64749e2394c84a75819a9f88d38f903a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:30:52.288795 17556 ts_tablet_manager.cc:1905] T a81c27cf5fc6482e8f92b8104f22d7fc P d68d1d5e4ad74b64b5e6b439a05d3fd7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 07:30:52.289388 17159 tablet_service.cc:1467] Processing CreateTablet for tablet 07ebfc03f86f4453929e77ca08c64435 (DEFAULT_TABLE table=test-table [id=360fa20582a1484ca4c00554d0b621ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:30:52.289467 17439 tablet_service.cc:1467] Processing CreateTablet for tablet 07ebfc03f86f4453929e77ca08c64435 (DEFAULT_TABLE table=test-table [id=360fa20582a1484ca4c00554d0b621ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:30:52.289525 17159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07ebfc03f86f4453929e77ca08c64435. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:30:52.289600 17439 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07ebfc03f86f4453929e77ca08c64435. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:30:52.289592 17299 tablet_service.cc:1467] Processing CreateTablet for tablet 07ebfc03f86f4453929e77ca08c64435 (DEFAULT_TABLE table=test-table [id=360fa20582a1484ca4c00554d0b621ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:30:52.289682 17299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07ebfc03f86f4453929e77ca08c64435. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:30:52.290611 17556 ts_tablet_manager.cc:1918] T a81c27cf5fc6482e8f92b8104f22d7fc P d68d1d5e4ad74b64b5e6b439a05d3fd7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 07:30:52.290660 17556 log.cc:1196] T a81c27cf5fc6482e8f92b8104f22d7fc P d68d1d5e4ad74b64b5e6b439a05d3fd7: Deleting WAL directory at /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790321392278483-9480-0/minicluster-data/ts-0/wal/wals/a81c27cf5fc6482e8f92b8104f22d7fc
I20260925 07:30:52.290771 17534 tablet_bootstrap.cc:492] T 07ebfc03f86f4453929e77ca08c64435 P d68d1d5e4ad74b64b5e6b439a05d3fd7: Bootstrap starting.
I20260925 07:30:52.290884 17556 ts_tablet_manager.cc:1939] T a81c27cf5fc6482e8f92b8104f22d7fc P d68d1d5e4ad74b64b5e6b439a05d3fd7: Deleting consensus metadata
I20260925 07:30:52.291107 17533 tablet_bootstrap.cc:492] T 07ebfc03f86f4453929e77ca08c64435 P f19ede8bffa7471e9146e45d3fadd8ce: Bootstrap starting.
I20260925 07:30:52.291157 17534 tablet_bootstrap.cc:654] T 07ebfc03f86f4453929e77ca08c64435 P d68d1d5e4ad74b64b5e6b439a05d3fd7: Neither blocks nor log segments found. Creating new log.
I20260925 07:30:52.291181 17555 ts_tablet_manager.cc:1905] T a81c27cf5fc6482e8f92b8104f22d7fc P f19ede8bffa7471e9146e45d3fadd8ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 07:30:52.291363 17533 tablet_bootstrap.cc:654] T 07ebfc03f86f4453929e77ca08c64435 P f19ede8bffa7471e9146e45d3fadd8ce: Neither blocks nor log segments found. Creating new log.
I20260925 07:30:52.291565 17538 tablet_bootstrap.cc:492] T 07ebfc03f86f4453929e77ca08c64435 P 64749e2394c84a75819a9f88d38f903a: Bootstrap starting.
I20260925 07:30:52.291767 17007 catalog_manager.cc:4841] TS d68d1d5e4ad74b64b5e6b439a05d3fd7 (127.9.66.1:39919): tablet a81c27cf5fc6482e8f92b8104f22d7fc (table test-table [id=c50023367a4a4fbe869fd4c41faa200e]) successfully deleted
I20260925 07:30:52.291989 17538 tablet_bootstrap.cc:654] T 07ebfc03f86f4453929e77ca08c64435 P 64749e2394c84a75819a9f88d38f903a: Neither blocks nor log segments found. Creating new log.
I20260925 07:30:52.292306 17533 tablet_bootstrap.cc:492] T 07ebfc03f86f4453929e77ca08c64435 P f19ede8bffa7471e9146e45d3fadd8ce: No bootstrap required, opened a new log
I20260925 07:30:52.292347 17533 ts_tablet_manager.cc:1397] T 07ebfc03f86f4453929e77ca08c64435 P f19ede8bffa7471e9146e45d3fadd8ce: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260925 07:30:52.292441 17538 tablet_bootstrap.cc:492] T 07ebfc03f86f4453929e77ca08c64435 P 64749e2394c84a75819a9f88d38f903a: No bootstrap required, opened a new log
I20260925 07:30:52.292488 17538 ts_tablet_manager.cc:1397] T 07ebfc03f86f4453929e77ca08c64435 P 64749e2394c84a75819a9f88d38f903a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 07:30:52.292487 17533 raft_consensus.cc:348] T 07ebfc03f86f4453929e77ca08c64435 P f19ede8bffa7471e9146e45d3fadd8ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d68d1d5e4ad74b64b5e6b439a05d3fd7" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 39919 } } peers { permanent_uuid: "64749e2394c84a75819a9f88d38f903a" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 33491 } } peers { permanent_uuid: "f19ede8bffa7471e9146e45d3fadd8ce" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 32865 } }
I20260925 07:30:52.292577 17533 raft_consensus.cc:374] T 07ebfc03f86f4453929e77ca08c64435 P f19ede8bffa7471e9146e45d3fadd8ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:30:52.292603 17533 raft_consensus.cc:729] T 07ebfc03f86f4453929e77ca08c64435 P f19ede8bffa7471e9146e45d3fadd8ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f19ede8bffa7471e9146e45d3fadd8ce, State: Initialized, Role: FOLLOWER
I20260925 07:30:52.292611 17538 raft_consensus.cc:348] T 07ebfc03f86f4453929e77ca08c64435 P 64749e2394c84a75819a9f88d38f903a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d68d1d5e4ad74b64b5e6b439a05d3fd7" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 39919 } } peers { permanent_uuid: "64749e2394c84a75819a9f88d38f903a" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 33491 } } peers { permanent_uuid: "f19ede8bffa7471e9146e45d3fadd8ce" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 32865 } }
I20260925 07:30:52.292644 17533 consensus_queue.cc:260] T 07ebfc03f86f4453929e77ca08c64435 P f19ede8bffa7471e9146e45d3fadd8ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d68d1d5e4ad74b64b5e6b439a05d3fd7" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 39919 } } peers { permanent_uuid: "64749e2394c84a75819a9f88d38f903a" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 33491 } } peers { permanent_uuid: "f19ede8bffa7471e9146e45d3fadd8ce" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 32865 } }
I20260925 07:30:52.292686 17538 raft_consensus.cc:374] T 07ebfc03f86f4453929e77ca08c64435 P 64749e2394c84a75819a9f88d38f903a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:30:52.292713 17538 raft_consensus.cc:729] T 07ebfc03f86f4453929e77ca08c64435 P 64749e2394c84a75819a9f88d38f903a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64749e2394c84a75819a9f88d38f903a, State: Initialized, Role: FOLLOWER
I20260925 07:30:52.292730 17533 ts_tablet_manager.cc:1428] T 07ebfc03f86f4453929e77ca08c64435 P f19ede8bffa7471e9146e45d3fadd8ce: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:52.292754 17538 consensus_queue.cc:260] T 07ebfc03f86f4453929e77ca08c64435 P 64749e2394c84a75819a9f88d38f903a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d68d1d5e4ad74b64b5e6b439a05d3fd7" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 39919 } } peers { permanent_uuid: "64749e2394c84a75819a9f88d38f903a" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 33491 } } peers { permanent_uuid: "f19ede8bffa7471e9146e45d3fadd8ce" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 32865 } }
I20260925 07:30:52.292857 17538 ts_tablet_manager.cc:1428] T 07ebfc03f86f4453929e77ca08c64435 P 64749e2394c84a75819a9f88d38f903a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:52.292881 17555 ts_tablet_manager.cc:1918] T a81c27cf5fc6482e8f92b8104f22d7fc P f19ede8bffa7471e9146e45d3fadd8ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 07:30:52.292641 17557 ts_tablet_manager.cc:1905] T a81c27cf5fc6482e8f92b8104f22d7fc P 64749e2394c84a75819a9f88d38f903a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 07:30:52.292918 17555 log.cc:1196] T a81c27cf5fc6482e8f92b8104f22d7fc P f19ede8bffa7471e9146e45d3fadd8ce: Deleting WAL directory at /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790321392278483-9480-0/minicluster-data/ts-2/wal/wals/a81c27cf5fc6482e8f92b8104f22d7fc
I20260925 07:30:52.293113 17555 ts_tablet_manager.cc:1939] T a81c27cf5fc6482e8f92b8104f22d7fc P f19ede8bffa7471e9146e45d3fadd8ce: Deleting consensus metadata
I20260925 07:30:52.293476 17007 catalog_manager.cc:4841] TS f19ede8bffa7471e9146e45d3fadd8ce (127.9.66.3:32865): tablet a81c27cf5fc6482e8f92b8104f22d7fc (table test-table [id=c50023367a4a4fbe869fd4c41faa200e]) successfully deleted
I20260925 07:30:52.293957 17534 tablet_bootstrap.cc:492] T 07ebfc03f86f4453929e77ca08c64435 P d68d1d5e4ad74b64b5e6b439a05d3fd7: No bootstrap required, opened a new log
I20260925 07:30:52.294006 17534 ts_tablet_manager.cc:1397] T 07ebfc03f86f4453929e77ca08c64435 P d68d1d5e4ad74b64b5e6b439a05d3fd7: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260925 07:30:52.294145 17534 raft_consensus.cc:348] T 07ebfc03f86f4453929e77ca08c64435 P d68d1d5e4ad74b64b5e6b439a05d3fd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d68d1d5e4ad74b64b5e6b439a05d3fd7" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 39919 } } peers { permanent_uuid: "64749e2394c84a75819a9f88d38f903a" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 33491 } } peers { permanent_uuid: "f19ede8bffa7471e9146e45d3fadd8ce" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 32865 } }
I20260925 07:30:52.294217 17534 raft_consensus.cc:374] T 07ebfc03f86f4453929e77ca08c64435 P d68d1d5e4ad74b64b5e6b439a05d3fd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:30:52.294250 17534 raft_consensus.cc:729] T 07ebfc03f86f4453929e77ca08c64435 P d68d1d5e4ad74b64b5e6b439a05d3fd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d68d1d5e4ad74b64b5e6b439a05d3fd7, State: Initialized, Role: FOLLOWER
I20260925 07:30:52.294296 17534 consensus_queue.cc:260] T 07ebfc03f86f4453929e77ca08c64435 P d68d1d5e4ad74b64b5e6b439a05d3fd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d68d1d5e4ad74b64b5e6b439a05d3fd7" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 39919 } } peers { permanent_uuid: "64749e2394c84a75819a9f88d38f903a" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 33491 } } peers { permanent_uuid: "f19ede8bffa7471e9146e45d3fadd8ce" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 32865 } }
I20260925 07:30:52.294374 17534 ts_tablet_manager.cc:1428] T 07ebfc03f86f4453929e77ca08c64435 P d68d1d5e4ad74b64b5e6b439a05d3fd7: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 07:30:52.294428 17557 ts_tablet_manager.cc:1918] T a81c27cf5fc6482e8f92b8104f22d7fc P 64749e2394c84a75819a9f88d38f903a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 07:30:52.294483 17557 log.cc:1196] T a81c27cf5fc6482e8f92b8104f22d7fc P 64749e2394c84a75819a9f88d38f903a: Deleting WAL directory at /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790321392278483-9480-0/minicluster-data/ts-1/wal/wals/a81c27cf5fc6482e8f92b8104f22d7fc
I20260925 07:30:52.294688 17557 ts_tablet_manager.cc:1939] T a81c27cf5fc6482e8f92b8104f22d7fc P 64749e2394c84a75819a9f88d38f903a: Deleting consensus metadata
I20260925 07:30:52.294984 17007 catalog_manager.cc:4841] TS 64749e2394c84a75819a9f88d38f903a (127.9.66.2:33491): tablet a81c27cf5fc6482e8f92b8104f22d7fc (table test-table [id=c50023367a4a4fbe869fd4c41faa200e]) successfully deleted
I20260925 07:30:52.488595 17541 raft_consensus.cc:482] T 07ebfc03f86f4453929e77ca08c64435 P 64749e2394c84a75819a9f88d38f903a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 07:30:52.488725 17541 raft_consensus.cc:504] T 07ebfc03f86f4453929e77ca08c64435 P 64749e2394c84a75819a9f88d38f903a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d68d1d5e4ad74b64b5e6b439a05d3fd7" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 39919 } } peers { permanent_uuid: "64749e2394c84a75819a9f88d38f903a" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 33491 } } peers { permanent_uuid: "f19ede8bffa7471e9146e45d3fadd8ce" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 32865 } }
I20260925 07:30:52.488919 17541 leader_election.cc:288] T 07ebfc03f86f4453929e77ca08c64435 P 64749e2394c84a75819a9f88d38f903a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d68d1d5e4ad74b64b5e6b439a05d3fd7 (127.9.66.1:39919), f19ede8bffa7471e9146e45d3fadd8ce (127.9.66.3:32865)
I20260925 07:30:52.489156 17458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07ebfc03f86f4453929e77ca08c64435" candidate_uuid: "64749e2394c84a75819a9f88d38f903a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f19ede8bffa7471e9146e45d3fadd8ce" is_pre_election: true
I20260925 07:30:52.489154 17179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07ebfc03f86f4453929e77ca08c64435" candidate_uuid: "64749e2394c84a75819a9f88d38f903a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d68d1d5e4ad74b64b5e6b439a05d3fd7" is_pre_election: true
I20260925 07:30:52.489272 17179 raft_consensus.cc:2454] T 07ebfc03f86f4453929e77ca08c64435 P d68d1d5e4ad74b64b5e6b439a05d3fd7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64749e2394c84a75819a9f88d38f903a in term 0.
I20260925 07:30:52.489272 17458 raft_consensus.cc:2454] T 07ebfc03f86f4453929e77ca08c64435 P f19ede8bffa7471e9146e45d3fadd8ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64749e2394c84a75819a9f88d38f903a in term 0.
I20260925 07:30:52.489449 17251 leader_election.cc:302] T 07ebfc03f86f4453929e77ca08c64435 P 64749e2394c84a75819a9f88d38f903a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64749e2394c84a75819a9f88d38f903a, d68d1d5e4ad74b64b5e6b439a05d3fd7; no voters:
I20260925 07:30:52.489579 17541 raft_consensus.cc:2781] T 07ebfc03f86f4453929e77ca08c64435 P 64749e2394c84a75819a9f88d38f903a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 07:30:52.489621 17541 raft_consensus.cc:482] T 07ebfc03f86f4453929e77ca08c64435 P 64749e2394c84a75819a9f88d38f903a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 07:30:52.489641 17541 raft_consensus.cc:3037] T 07ebfc03f86f4453929e77ca08c64435 P 64749e2394c84a75819a9f88d38f903a [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:30:52.490325 17541 raft_consensus.cc:504] T 07ebfc03f86f4453929e77ca08c64435 P 64749e2394c84a75819a9f88d38f903a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d68d1d5e4ad74b64b5e6b439a05d3fd7" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 39919 } } peers { permanent_uuid: "64749e2394c84a75819a9f88d38f903a" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 33491 } } peers { permanent_uuid: "f19ede8bffa7471e9146e45d3fadd8ce" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 32865 } }
I20260925 07:30:52.490455 17541 leader_election.cc:288] T 07ebfc03f86f4453929e77ca08c64435 P 64749e2394c84a75819a9f88d38f903a [CANDIDATE]: Term 1 election: Requested vote from peers d68d1d5e4ad74b64b5e6b439a05d3fd7 (127.9.66.1:39919), f19ede8bffa7471e9146e45d3fadd8ce (127.9.66.3:32865)
I20260925 07:30:52.490615 17179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07ebfc03f86f4453929e77ca08c64435" candidate_uuid: "64749e2394c84a75819a9f88d38f903a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d68d1d5e4ad74b64b5e6b439a05d3fd7"
I20260925 07:30:52.490705 17179 raft_consensus.cc:3037] T 07ebfc03f86f4453929e77ca08c64435 P d68d1d5e4ad74b64b5e6b439a05d3fd7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:30:52.490657 17458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07ebfc03f86f4453929e77ca08c64435" candidate_uuid: "64749e2394c84a75819a9f88d38f903a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f19ede8bffa7471e9146e45d3fadd8ce"
I20260925 07:30:52.490747 17458 raft_consensus.cc:3037] T 07ebfc03f86f4453929e77ca08c64435 P f19ede8bffa7471e9146e45d3fadd8ce [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:30:52.491305 17179 raft_consensus.cc:2454] T 07ebfc03f86f4453929e77ca08c64435 P d68d1d5e4ad74b64b5e6b439a05d3fd7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64749e2394c84a75819a9f88d38f903a in term 1.
I20260925 07:30:52.491432 17458 raft_consensus.cc:2454] T 07ebfc03f86f4453929e77ca08c64435 P f19ede8bffa7471e9146e45d3fadd8ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64749e2394c84a75819a9f88d38f903a in term 1.
I20260925 07:30:52.491501 17251 leader_election.cc:302] T 07ebfc03f86f4453929e77ca08c64435 P 64749e2394c84a75819a9f88d38f903a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64749e2394c84a75819a9f88d38f903a, d68d1d5e4ad74b64b5e6b439a05d3fd7; no voters:
I20260925 07:30:52.491602 17541 raft_consensus.cc:2781] T 07ebfc03f86f4453929e77ca08c64435 P 64749e2394c84a75819a9f88d38f903a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:30:52.491655 17541 raft_consensus.cc:686] T 07ebfc03f86f4453929e77ca08c64435 P 64749e2394c84a75819a9f88d38f903a [term 1 LEADER]: Becoming Leader. State: Replica: 64749e2394c84a75819a9f88d38f903a, State: Running, Role: LEADER
I20260925 07:30:52.491700 17541 consensus_queue.cc:237] T 07ebfc03f86f4453929e77ca08c64435 P 64749e2394c84a75819a9f88d38f903a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d68d1d5e4ad74b64b5e6b439a05d3fd7" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 39919 } } peers { permanent_uuid: "64749e2394c84a75819a9f88d38f903a" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 33491 } } peers { permanent_uuid: "f19ede8bffa7471e9146e45d3fadd8ce" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 32865 } }
I20260925 07:30:52.492295 17021 catalog_manager.cc:5495] T 07ebfc03f86f4453929e77ca08c64435 P 64749e2394c84a75819a9f88d38f903a reported cstate change: term changed from 0 to 1, leader changed from <none> to 64749e2394c84a75819a9f88d38f903a (127.9.66.2). New cstate: current_term: 1 leader_uuid: "64749e2394c84a75819a9f88d38f903a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d68d1d5e4ad74b64b5e6b439a05d3fd7" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 39919 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64749e2394c84a75819a9f88d38f903a" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 33491 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f19ede8bffa7471e9146e45d3fadd8ce" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 32865 } health_report { overall_health: UNKNOWN } } }
I20260925 07:30:52.523178 17532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:52.521261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55302 (local address 127.9.66.2:33491)
0925 07:30:52.521295 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:52.521297 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:52.521305 (+ 8us) server_negotiation.cc:407] Connection header received
0925 07:30:52.521360 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:52.521360 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:52.521369 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:52.521412 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 07:30:52.521494 (+ 82us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:52.521954 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:52.522697 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:52.522870 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:52.522921 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:52.523012 (+ 91us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 07:30:52.523035 (+ 23us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 07:30:52.523068 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:52.523111 (+ 43us) server_negotiation.cc:299] Negotiation successful
0925 07:30:52.523124 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260925 07:30:52.525280 17458 raft_consensus.cc:1261] T 07ebfc03f86f4453929e77ca08c64435 P f19ede8bffa7471e9146e45d3fadd8ce [term 1 FOLLOWER]: Refusing update from remote peer 64749e2394c84a75819a9f88d38f903a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:30:52.525287 17179 raft_consensus.cc:1261] T 07ebfc03f86f4453929e77ca08c64435 P d68d1d5e4ad74b64b5e6b439a05d3fd7 [term 1 FOLLOWER]: Refusing update from remote peer 64749e2394c84a75819a9f88d38f903a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:30:52.525478 17563 consensus_queue.cc:1035] T 07ebfc03f86f4453929e77ca08c64435 P 64749e2394c84a75819a9f88d38f903a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d68d1d5e4ad74b64b5e6b439a05d3fd7" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 39919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:30:52.525611 17580 consensus_queue.cc:1035] T 07ebfc03f86f4453929e77ca08c64435 P 64749e2394c84a75819a9f88d38f903a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f19ede8bffa7471e9146e45d3fadd8ce" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 32865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:30:52.535274 17021 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50642:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 07:30:52.535365 17021 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50642:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 07:30:52.535986 17021 catalog_manager.cc:5782] T 00000000000000000000000000000000 P df6bdd3c1abe434eb57ce7f7b79dd302: Sending DeleteTablet for 3 replicas of tablet 07ebfc03f86f4453929e77ca08c64435
I20260925 07:30:52.536249 17439 tablet_service.cc:1514] Processing DeleteTablet for tablet 07ebfc03f86f4453929e77ca08c64435 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:30:52 UTC) from {username='oryx', principal='oryx/127.9.66.62@KRBTEST.COM'} at 127.0.0.1:35462
I20260925 07:30:52.536273 17159 tablet_service.cc:1514] Processing DeleteTablet for tablet 07ebfc03f86f4453929e77ca08c64435 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:30:52 UTC) from {username='oryx', principal='oryx/127.9.66.62@KRBTEST.COM'} at 127.0.0.1:50720
I20260925 07:30:52.536278 17299 tablet_service.cc:1514] Processing DeleteTablet for tablet 07ebfc03f86f4453929e77ca08c64435 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:30:52 UTC) from {username='oryx', principal='oryx/127.9.66.62@KRBTEST.COM'} at 127.0.0.1:55282
I20260925 07:30:52.536337 17555 tablet_replica.cc:330] T 07ebfc03f86f4453929e77ca08c64435 P f19ede8bffa7471e9146e45d3fadd8ce: stopping tablet replica
I20260925 07:30:52.536360 17557 tablet_replica.cc:330] T 07ebfc03f86f4453929e77ca08c64435 P 64749e2394c84a75819a9f88d38f903a: stopping tablet replica
I20260925 07:30:52.536391 17555 raft_consensus.cc:2229] T 07ebfc03f86f4453929e77ca08c64435 P f19ede8bffa7471e9146e45d3fadd8ce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 07:30:52.536410 17557 raft_consensus.cc:2229] T 07ebfc03f86f4453929e77ca08c64435 P 64749e2394c84a75819a9f88d38f903a [term 1 LEADER]: Raft consensus shutting down.
I20260925 07:30:52.536423 17555 raft_consensus.cc:2258] T 07ebfc03f86f4453929e77ca08c64435 P f19ede8bffa7471e9146e45d3fadd8ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:30:52.536484 17557 raft_consensus.cc:2258] T 07ebfc03f86f4453929e77ca08c64435 P 64749e2394c84a75819a9f88d38f903a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:30:52.536567 17556 tablet_replica.cc:330] T 07ebfc03f86f4453929e77ca08c64435 P d68d1d5e4ad74b64b5e6b439a05d3fd7: stopping tablet replica
I20260925 07:30:52.536628 17556 raft_consensus.cc:2229] T 07ebfc03f86f4453929e77ca08c64435 P d68d1d5e4ad74b64b5e6b439a05d3fd7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 07:30:52.536669 17556 raft_consensus.cc:2258] T 07ebfc03f86f4453929e77ca08c64435 P d68d1d5e4ad74b64b5e6b439a05d3fd7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:30:52.536818 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 17094
I20260925 07:30:52.536823 17555 ts_tablet_manager.cc:1905] T 07ebfc03f86f4453929e77ca08c64435 P f19ede8bffa7471e9146e45d3fadd8ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 07:30:52.536952 17556 ts_tablet_manager.cc:1905] T 07ebfc03f86f4453929e77ca08c64435 P d68d1d5e4ad74b64b5e6b439a05d3fd7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 07:30:52.538546 17555 ts_tablet_manager.cc:1918] T 07ebfc03f86f4453929e77ca08c64435 P f19ede8bffa7471e9146e45d3fadd8ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 07:30:52.538617 17555 log.cc:1196] T 07ebfc03f86f4453929e77ca08c64435 P f19ede8bffa7471e9146e45d3fadd8ce: Deleting WAL directory at /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790321392278483-9480-0/minicluster-data/ts-2/wal/wals/07ebfc03f86f4453929e77ca08c64435
I20260925 07:30:52.538851 17555 ts_tablet_manager.cc:1939] T 07ebfc03f86f4453929e77ca08c64435 P f19ede8bffa7471e9146e45d3fadd8ce: Deleting consensus metadata
I20260925 07:30:52.539065 17557 ts_tablet_manager.cc:1905] T 07ebfc03f86f4453929e77ca08c64435 P 64749e2394c84a75819a9f88d38f903a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 07:30:52.539176 17007 catalog_manager.cc:4841] TS f19ede8bffa7471e9146e45d3fadd8ce (127.9.66.3:32865): tablet 07ebfc03f86f4453929e77ca08c64435 (table test-table [id=360fa20582a1484ca4c00554d0b621ec]) successfully deleted
I20260925 07:30:52.540162 17557 ts_tablet_manager.cc:1918] T 07ebfc03f86f4453929e77ca08c64435 P 64749e2394c84a75819a9f88d38f903a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 07:30:52.540199 17557 log.cc:1196] T 07ebfc03f86f4453929e77ca08c64435 P 64749e2394c84a75819a9f88d38f903a: Deleting WAL directory at /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790321392278483-9480-0/minicluster-data/ts-1/wal/wals/07ebfc03f86f4453929e77ca08c64435
I20260925 07:30:52.540349 17557 ts_tablet_manager.cc:1939] T 07ebfc03f86f4453929e77ca08c64435 P 64749e2394c84a75819a9f88d38f903a: Deleting consensus metadata
I20260925 07:30:52.540735 17007 catalog_manager.cc:4841] TS 64749e2394c84a75819a9f88d38f903a (127.9.66.2:33491): tablet 07ebfc03f86f4453929e77ca08c64435 (table test-table [id=360fa20582a1484ca4c00554d0b621ec]) successfully deleted
W20260925 07:30:52.542661 17007 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.66.1:39919 (error 108)
I20260925 07:30:52.542898 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 17234
I20260925 07:30:52.543149 17617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:52.542974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.1:39919 (local address 127.0.0.1:50730)
0925 07:30:52.543062 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:52.543100 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.1:39919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72,"thread_start_us":49,"threads_started":1}
W20260925 07:30:52.543463 17007 catalog_manager.cc:4584] TS d68d1d5e4ad74b64b5e6b439a05d3fd7 (127.9.66.1:39919): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 07ebfc03f86f4453929e77ca08c64435: Network error: Client connection negotiation failed: client connection to 127.9.66.1:39919: connect: Connection refused (error 111)
I20260925 07:30:52.549325 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 17374
I20260925 07:30:52.555061 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 16990
2026-09-25T07:30:52Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3034 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790321392278483-9480-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 07:30:52 dist-test-slave-57fn krb5kdc[17620](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 07:30:52 dist-test-slave-57fn krb5kdc[17620](info): set up 2 sockets
Sep 25 07:30:52 dist-test-slave-57fn krb5kdc[17620](info): commencing operation
krb5kdc: starting...
W20260925 07:30:53.574154 9480 mini_kdc.cc:121] Time spent starting KDC: real 0.998s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790321392278483-9480-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790321392278483-9480-0/krb5kdc/test-user.keytab.
Sep 25 07:30:53 dist-test-slave-57fn krb5kdc[17620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321453, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T07:30:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T07:30:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.66.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.66.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.66.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab.
Entry for principal kudu/127.9.66.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab.
I20260925 07:30:53.689168 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790321392278483-9480-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790321392278483-9480-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790321392278483-9480-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.66.62:44369
--webserver_interface=127.9.66.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab
--principal=kudu/127.9.66.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:38787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.66.62:44369
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:30:53.766973 17633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:30:53.767215 17633 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:30:53.767251 17633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:30:53.768381 17633 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:30:53.768445 17633 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:30:53.768468 17633 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:30:53.768491 17633 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:30:53.768517 17633 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:30:53.770059 17633 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:38787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790321392278483-9480-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790321392278483-9480-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.66.62:44369
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.66.62:44369
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.66.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17633
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790321392278483-9480-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:30:53.770355 17633 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:30:53.770567 17633 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:30:53.775319 17641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:30:53.775347 17638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:30:53.775388 17639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:30:53.775435 17633 server_base.cc:1034] running on GCE node
I20260925 07:30:53.775745 17633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:30:53.775995 17633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:30:53.777139 17633 hybrid_clock.cc:648] HybridClock initialized: now 1790321453777128 us; error 31 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-25T07:30:53Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1230 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 07:30:53 dist-test-slave-57fn krb5kdc[17646](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 07:30:53 dist-test-slave-57fn krb5kdc[17646](info): set up 2 sockets
Sep 25 07:30:53 dist-test-slave-57fn krb5kdc[17646](info): commencing operation
krb5kdc: starting...
W20260925 07:30:54.814059 9480 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/krb5kdc/test-user.keytab.
Sep 25 07:30:54 dist-test-slave-57fn krb5kdc[17646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321454, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T07:30:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T07:30:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.66.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.66.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.66.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab.
Entry for principal kudu/127.9.66.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab.
I20260925 07:30:54.926285 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.66.62:32879
--webserver_interface=127.9.66.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab
--principal=kudu/127.9.66.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:42819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.66.62:32879
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:30:55.003729 17659 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:30:55.003960 17659 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:30:55.003997 17659 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:30:55.005126 17659 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:30:55.005188 17659 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:30:55.005213 17659 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:30:55.005234 17659 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:30:55.005261 17659 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:30:55.006714 17659 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:42819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.66.62:32879
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab
--principal=kudu/127.9.66.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.66.62:32879
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.66.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17659
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:30:55.006973 17659 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:30:55.007203 17659 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:30:55.011869 17665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:30:55.011900 17667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:30:55.012001 17659 server_base.cc:1034] running on GCE node
W20260925 07:30:55.011869 17664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:30:55.012306 17659 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:30:55.012573 17659 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:30:55.013723 17659 hybrid_clock.cc:648] HybridClock initialized: now 1790321455013729 us; error 48 us; skew 500 ppm
Sep 25 07:30:55 dist-test-slave-57fn krb5kdc[17646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321455, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:30:55.016062 17659 init.cc:377] Logged in from keytab as kudu/127.9.66.62@KRBTEST.COM (short username kudu)
I20260925 07:30:55.016474 17659 webserver.cc:460] Webserver started at http://127.9.66.62:41705/ using document root <none> and password file <none>
I20260925 07:30:55.016644 17659 fs_manager.cc:359] Metadata directory not provided
I20260925 07:30:55.016683 17659 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:30:55.016765 17659 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:30:55.017781 17659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/minicluster-data/master-0/data/instance:
uuid: "aaf5cbfc86744cd1956ea2aa4645a979"
format_stamp: "Formatted at 2026-09-25 07:30:55 on dist-test-slave-57fn"
server_key: "0a094162d18b77073a1e115655411fbb"
server_key_iv: "7d5963d4a2f2c5024955ff502f79b7da"
server_key_version: "encryptionkey@0"
I20260925 07:30:55.018059 17659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/minicluster-data/master-0/wal/instance:
uuid: "aaf5cbfc86744cd1956ea2aa4645a979"
format_stamp: "Formatted at 2026-09-25 07:30:55 on dist-test-slave-57fn"
server_key: "0a094162d18b77073a1e115655411fbb"
server_key_iv: "7d5963d4a2f2c5024955ff502f79b7da"
server_key_version: "encryptionkey@0"
I20260925 07:30:55.019222 17659 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:30:55.019874 17674 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:55.020057 17659 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 07:30:55.020139 17659 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/minicluster-data/master-0/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/minicluster-data/master-0/wal
uuid: "aaf5cbfc86744cd1956ea2aa4645a979"
format_stamp: "Formatted at 2026-09-25 07:30:55 on dist-test-slave-57fn"
server_key: "0a094162d18b77073a1e115655411fbb"
server_key_iv: "7d5963d4a2f2c5024955ff502f79b7da"
server_key_version: "encryptionkey@0"
I20260925 07:30:55.020196 17659 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:30:55.032241 17659 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:30:55.033943 17659 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:30:55.034046 17659 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:30:55.037655 17659 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.62:32879
I20260925 07:30:55.037714 17726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.62:32879 every 8 connection(s)
I20260925 07:30:55.038013 17659 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
I20260925 07:30:55.038826 17727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:30:55.040401 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 17659
I20260925 07:30:55.040501 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/minicluster-data/master-0/wal/instance
I20260925 07:30:55.040685 9480 external_mini_cluster.cc:1428] Setting key 20236b48fba15d2d10343b7c7f6b3591
I20260925 07:30:55.041335 17727 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aaf5cbfc86744cd1956ea2aa4645a979: Bootstrap starting.
I20260925 07:30:55.041966 17727 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aaf5cbfc86744cd1956ea2aa4645a979: Neither blocks nor log segments found. Creating new log.
I20260925 07:30:55.042313 17727 log.cc:824] T 00000000000000000000000000000000 P aaf5cbfc86744cd1956ea2aa4645a979: Log is configured to *not* fsync() on all Append() calls
I20260925 07:30:55.043015 17727 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aaf5cbfc86744cd1956ea2aa4645a979: No bootstrap required, opened a new log
I20260925 07:30:55.045109 17727 raft_consensus.cc:348] T 00000000000000000000000000000000 P aaf5cbfc86744cd1956ea2aa4645a979 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaf5cbfc86744cd1956ea2aa4645a979" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 32879 } }
I20260925 07:30:55.045233 17727 raft_consensus.cc:374] T 00000000000000000000000000000000 P aaf5cbfc86744cd1956ea2aa4645a979 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:30:55.045253 17727 raft_consensus.cc:729] T 00000000000000000000000000000000 P aaf5cbfc86744cd1956ea2aa4645a979 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aaf5cbfc86744cd1956ea2aa4645a979, State: Initialized, Role: FOLLOWER
I20260925 07:30:55.045393 17727 consensus_queue.cc:260] T 00000000000000000000000000000000 P aaf5cbfc86744cd1956ea2aa4645a979 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaf5cbfc86744cd1956ea2aa4645a979" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 32879 } }
I20260925 07:30:55.045469 17727 raft_consensus.cc:388] T 00000000000000000000000000000000 P aaf5cbfc86744cd1956ea2aa4645a979 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 07:30:55.045501 17727 raft_consensus.cc:482] T 00000000000000000000000000000000 P aaf5cbfc86744cd1956ea2aa4645a979 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 07:30:55.045588 17727 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aaf5cbfc86744cd1956ea2aa4645a979 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:30:55.046286 17727 raft_consensus.cc:504] T 00000000000000000000000000000000 P aaf5cbfc86744cd1956ea2aa4645a979 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaf5cbfc86744cd1956ea2aa4645a979" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 32879 } }
I20260925 07:30:55.046404 17727 leader_election.cc:302] T 00000000000000000000000000000000 P aaf5cbfc86744cd1956ea2aa4645a979 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aaf5cbfc86744cd1956ea2aa4645a979; no voters:
I20260925 07:30:55.046586 17727 leader_election.cc:288] T 00000000000000000000000000000000 P aaf5cbfc86744cd1956ea2aa4645a979 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 07:30:55.046624 17732 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aaf5cbfc86744cd1956ea2aa4645a979 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:30:55.046819 17732 raft_consensus.cc:686] T 00000000000000000000000000000000 P aaf5cbfc86744cd1956ea2aa4645a979 [term 1 LEADER]: Becoming Leader. State: Replica: aaf5cbfc86744cd1956ea2aa4645a979, State: Running, Role: LEADER
I20260925 07:30:55.046844 17727 sys_catalog.cc:564] T 00000000000000000000000000000000 P aaf5cbfc86744cd1956ea2aa4645a979 [sys.catalog]: configured and running, proceeding with master startup.
Sep 25 07:30:55 dist-test-slave-57fn krb5kdc[17646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321454, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.66.62@KRBTEST.COM
I20260925 07:30:55.048111 17732 consensus_queue.cc:237] T 00000000000000000000000000000000 P aaf5cbfc86744cd1956ea2aa4645a979 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaf5cbfc86744cd1956ea2aa4645a979" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 32879 } }
I20260925 07:30:55.048647 17734 sys_catalog.cc:455] T 00000000000000000000000000000000 P aaf5cbfc86744cd1956ea2aa4645a979 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aaf5cbfc86744cd1956ea2aa4645a979" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaf5cbfc86744cd1956ea2aa4645a979" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 32879 } } }
I20260925 07:30:55.048743 17734 sys_catalog.cc:458] T 00000000000000000000000000000000 P aaf5cbfc86744cd1956ea2aa4645a979 [sys.catalog]: This master's current role is: LEADER
I20260925 07:30:55.048858 17733 sys_catalog.cc:455] T 00000000000000000000000000000000 P aaf5cbfc86744cd1956ea2aa4645a979 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aaf5cbfc86744cd1956ea2aa4645a979. Latest consensus state: current_term: 1 leader_uuid: "aaf5cbfc86744cd1956ea2aa4645a979" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaf5cbfc86744cd1956ea2aa4645a979" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 32879 } } }
I20260925 07:30:55.048945 17733 sys_catalog.cc:458] T 00000000000000000000000000000000 P aaf5cbfc86744cd1956ea2aa4645a979 [sys.catalog]: This master's current role is: LEADER
W20260925 07:30:55.049736 17748 catalog_manager.cc:1558] T 00000000000000000000000000000000 P aaf5cbfc86744cd1956ea2aa4645a979: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 07:30:55.050606 17748 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 07:30:55.050720 17746 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 07:30:55.050897 17746 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 07:30:55.051905 17730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:55.041173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46934 (local address 127.9.66.62:32879)
0925 07:30:55.041332 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:55.041339 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:55.041361 (+ 22us) server_negotiation.cc:407] Connection header received
0925 07:30:55.041875 (+ 514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:55.041882 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:55.041925 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:55.042076 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:30:55.042838 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:55.043763 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:55.044902 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:55.045318 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:55.045429 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:55.048257 (+ 2828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:30:55.048261 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:30:55.048261 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:30:55.048272 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:30:55.050279 (+ 2007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:55.050700 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:55.050704 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:55.050704 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:55.050802 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:55.051014 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:55.051014 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:55.051015 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:55.051302 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:30:55.051435 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:55.051667 (+ 232us) server_negotiation.cc:299] Negotiation successful
0925 07:30:55.051777 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":86,"threads_started":1}
I20260925 07:30:55.052495 17746 catalog_manager.cc:1347] Generated new cluster ID: bc747919f91e49cab589258a827efa7f
I20260925 07:30:55.052544 17746 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 07:30:55.080142 17746 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 07:30:55.080744 17746 catalog_manager.cc:1504] Loading token signing keys...
I20260925 07:30:55.088967 17746 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aaf5cbfc86744cd1956ea2aa4645a979: Generated new TSK 0
I20260925 07:30:55.089161 17746 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.66.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.66.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.66.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.1.keytab.
Entry for principal kudu/127.9.66.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.1.keytab.
I20260925 07:30:55.118307 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.66.1:0
--local_ip_for_outbound_sockets=127.9.66.1
--webserver_interface=127.9.66.1
--webserver_port=0
--tserver_master_addrs=127.9.66.62:32879
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.1.keytab
--principal=kudu/127.9.66.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:42819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:30:55.195479 17753 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:30:55.195662 17753 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:30:55.195685 17753 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:30:55.196744 17753 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:30:55.196779 17753 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:30:55.196803 17753 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.66.1
I20260925 07:30:55.198180 17753 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:42819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.1.keytab
--principal=kudu/127.9.66.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.66.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.66.1
--webserver_port=0
--tserver_master_addrs=127.9.66.62:32879
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17753
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.66.1
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:30:55.198339 17753 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:30:55.198495 17753 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:30:55.203202 17759 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:30:55.203251 17753 server_base.cc:1034] running on GCE node
W20260925 07:30:55.203230 17761 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:30:55.203217 17758 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:30:55.203554 17753 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:30:55.203744 17753 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:30:55.204869 17753 hybrid_clock.cc:648] HybridClock initialized: now 1790321455204852 us; error 34 us; skew 500 ppm
Sep 25 07:30:55 dist-test-slave-57fn krb5kdc[17646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321455, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:30:55.207284 17753 init.cc:377] Logged in from keytab as kudu/127.9.66.1@KRBTEST.COM (short username kudu)
I20260925 07:30:55.207674 17753 webserver.cc:460] Webserver started at http://127.9.66.1:38313/ using document root <none> and password file <none>
I20260925 07:30:55.207891 17753 fs_manager.cc:359] Metadata directory not provided
I20260925 07:30:55.207935 17753 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:30:55.208096 17753 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:30:55.209147 17753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/minicluster-data/ts-0/data/instance:
uuid: "f858a8705331410d9a1703be45bf92b2"
format_stamp: "Formatted at 2026-09-25 07:30:55 on dist-test-slave-57fn"
server_key: "d780e833a18f69bda16e7d506d674477"
server_key_iv: "206156eb9c70fecf4fdd53fe2b98d10d"
server_key_version: "encryptionkey@0"
I20260925 07:30:55.209501 17753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/minicluster-data/ts-0/wal/instance:
uuid: "f858a8705331410d9a1703be45bf92b2"
format_stamp: "Formatted at 2026-09-25 07:30:55 on dist-test-slave-57fn"
server_key: "d780e833a18f69bda16e7d506d674477"
server_key_iv: "206156eb9c70fecf4fdd53fe2b98d10d"
server_key_version: "encryptionkey@0"
I20260925 07:30:55.210738 17753 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:30:55.211387 17768 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:55.211540 17753 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 07:30:55.211620 17753 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/minicluster-data/ts-0/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/minicluster-data/ts-0/wal
uuid: "f858a8705331410d9a1703be45bf92b2"
format_stamp: "Formatted at 2026-09-25 07:30:55 on dist-test-slave-57fn"
server_key: "d780e833a18f69bda16e7d506d674477"
server_key_iv: "206156eb9c70fecf4fdd53fe2b98d10d"
server_key_version: "encryptionkey@0"
I20260925 07:30:55.211669 17753 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:30:55.228674 17753 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:30:55.230738 17753 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:30:55.230840 17753 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:30:55.230996 17753 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:30:55.231278 17753 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:30:55.231310 17753 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:55.231330 17753 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:30:55.231372 17753 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:55.236757 17753 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.1:41339
I20260925 07:30:55.236820 17881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.1:41339 every 8 connection(s)
I20260925 07:30:55.237125 17753 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/minicluster-data/ts-0/data/info.pb
I20260925 07:30:55.242414 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 17753
I20260925 07:30:55.242532 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/minicluster-data/ts-0/wal/instance
I20260925 07:30:55.242730 9480 external_mini_cluster.cc:1428] Setting key fdaac2198ba543978b44577a474d6e5d
Sep 25 07:30:55 dist-test-slave-57fn krb5kdc[17646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321455, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.1@KRBTEST.COM for kudu/127.9.66.62@KRBTEST.COM
I20260925 07:30:55.246562 17884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:55.237783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:32879 (local address 127.9.66.1:48765)
0925 07:30:55.237928 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:55.237959 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0925 07:30:55.238379 (+ 420us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:30:55.238557 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:30:55.238560 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:30:55.239080 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:30:55.239251 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:55.239255 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:55.239900 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:55.239900 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:55.240721 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:55.240724 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:55.241046 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:55.241047 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:55.241139 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:55.241597 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:30:55.241606 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:30:55.243256 (+ 1650us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:30:55.245105 (+ 1849us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:30:55.245112 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:30:55.245115 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:30:55.245432 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:30:55.245672 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:30:55.245673 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:30:55.245673 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:30:55.245809 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:30:55.246033 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:30:55.246038 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:30:55.246199 (+ 161us) client_negotiation.cc:769] Sending connection context
0925 07:30:55.246285 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 07:30:55.246404 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":78,"threads_started":1}
I20260925 07:30:55.246562 17730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:55.237795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.1:48765 (local address 127.9.66.62:32879)
0925 07:30:55.237840 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:55.237843 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:55.238032 (+ 189us) server_negotiation.cc:407] Connection header received
0925 07:30:55.238460 (+ 428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:55.238461 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:55.238470 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:55.238510 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:30:55.239320 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:55.239803 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:55.240790 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:55.240981 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:55.241030 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:55.243336 (+ 2306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:30:55.243337 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:30:55.243337 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:30:55.243357 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:30:55.244996 (+ 1639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:55.245538 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:55.245539 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:55.245540 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:55.245599 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:55.245874 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:55.245875 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:55.245875 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:55.245974 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:30:55.246022 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:55.246324 (+ 302us) server_negotiation.cc:299] Negotiation successful
0925 07:30:55.246421 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 07:30:55.247118 17882 heartbeater.cc:347] Connected to a master server at 127.9.66.62:32879
I20260925 07:30:55.247220 17882 heartbeater.cc:464] Registering TS with master...
I20260925 07:30:55.247422 17882 heartbeater.cc:511] Master 127.9.66.62:32879 requested a full tablet report, sending...
I20260925 07:30:55.248037 17691 ts_manager.cc:194] Registered new tserver with Master: f858a8705331410d9a1703be45bf92b2 (127.9.66.1:41339)
I20260925 07:30:55.249049 17691 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.66.1@KRBTEST.COM'} at 127.9.66.1:48765
WARNING: no policy specified for kudu/127.9.66.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.66.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.66.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.2.keytab.
Entry for principal kudu/127.9.66.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.2.keytab.
I20260925 07:30:55.267745 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.66.2:0
--local_ip_for_outbound_sockets=127.9.66.2
--webserver_interface=127.9.66.2
--webserver_port=0
--tserver_master_addrs=127.9.66.62:32879
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.2.keytab
--principal=kudu/127.9.66.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:42819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:30:55.347131 17887 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:30:55.347332 17887 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:30:55.347375 17887 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:30:55.348554 17887 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:30:55.348591 17887 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:30:55.348646 17887 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.66.2
I20260925 07:30:55.350046 17887 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:42819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.2.keytab
--principal=kudu/127.9.66.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.66.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.66.2
--webserver_port=0
--tserver_master_addrs=127.9.66.62:32879
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17887
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.66.2
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:30:55.350265 17887 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:30:55.350471 17887 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:30:55.355159 17893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:30:55.355257 17887 server_base.cc:1034] running on GCE node
W20260925 07:30:55.355170 17892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:30:55.355159 17895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:30:55.355506 17887 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:30:55.355753 17887 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:30:55.356917 17887 hybrid_clock.cc:648] HybridClock initialized: now 1790321455356897 us; error 34 us; skew 500 ppm
Sep 25 07:30:55 dist-test-slave-57fn krb5kdc[17646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321455, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:30:55.359280 17887 init.cc:377] Logged in from keytab as kudu/127.9.66.2@KRBTEST.COM (short username kudu)
I20260925 07:30:55.359699 17887 webserver.cc:460] Webserver started at http://127.9.66.2:35039/ using document root <none> and password file <none>
I20260925 07:30:55.359903 17887 fs_manager.cc:359] Metadata directory not provided
I20260925 07:30:55.359967 17887 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:30:55.360107 17887 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:30:55.361096 17887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/minicluster-data/ts-1/data/instance:
uuid: "c2ba107511fb4aa88bdebfb7dd7c1078"
format_stamp: "Formatted at 2026-09-25 07:30:55 on dist-test-slave-57fn"
server_key: "8a4be71a1b1f32557df81e3e52089927"
server_key_iv: "859c53402400fad32e356e910373d6c1"
server_key_version: "encryptionkey@0"
I20260925 07:30:55.361438 17887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/minicluster-data/ts-1/wal/instance:
uuid: "c2ba107511fb4aa88bdebfb7dd7c1078"
format_stamp: "Formatted at 2026-09-25 07:30:55 on dist-test-slave-57fn"
server_key: "8a4be71a1b1f32557df81e3e52089927"
server_key_iv: "859c53402400fad32e356e910373d6c1"
server_key_version: "encryptionkey@0"
I20260925 07:30:55.362684 17887 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 07:30:55.363354 17902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:55.363507 17887 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 07:30:55.363602 17887 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/minicluster-data/ts-1/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/minicluster-data/ts-1/wal
uuid: "c2ba107511fb4aa88bdebfb7dd7c1078"
format_stamp: "Formatted at 2026-09-25 07:30:55 on dist-test-slave-57fn"
server_key: "8a4be71a1b1f32557df81e3e52089927"
server_key_iv: "859c53402400fad32e356e910373d6c1"
server_key_version: "encryptionkey@0"
I20260925 07:30:55.363675 17887 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:30:55.375174 17887 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:30:55.377326 17887 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:30:55.377471 17887 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:30:55.377702 17887 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:30:55.377951 17887 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:30:55.377979 17887 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:55.378000 17887 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:30:55.378018 17887 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:55.383677 17887 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.2:35345
I20260925 07:30:55.383749 18015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.2:35345 every 8 connection(s)
I20260925 07:30:55.384048 17887 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/minicluster-data/ts-1/data/info.pb
Sep 25 07:30:55 dist-test-slave-57fn krb5kdc[17646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321455, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.2@KRBTEST.COM for kudu/127.9.66.62@KRBTEST.COM
I20260925 07:30:55.392067 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 17887
I20260925 07:30:55.392187 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/minicluster-data/ts-1/wal/instance
I20260925 07:30:55.392412 9480 external_mini_cluster.cc:1428] Setting key a061cd303135187f57d234147822b30d
I20260925 07:30:55.392562 17730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:55.384737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.2:51887 (local address 127.9.66.62:32879)
0925 07:30:55.384793 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:55.384795 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:55.384979 (+ 184us) server_negotiation.cc:407] Connection header received
0925 07:30:55.385448 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:55.385449 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:55.385457 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:55.385553 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:30:55.386443 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:55.386946 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:55.387864 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:55.388034 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:55.388082 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:55.390033 (+ 1951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:30:55.390034 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:30:55.390034 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:30:55.390041 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:30:55.391335 (+ 1294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:55.391657 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:55.391658 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:55.391658 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:55.391703 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:55.391907 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:55.391908 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:55.391909 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:55.392046 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:30:55.392116 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:55.392397 (+ 281us) server_negotiation.cc:299] Negotiation successful
0925 07:30:55.392472 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260925 07:30:55.392750 18018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:55.384719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:32879 (local address 127.9.66.2:51887)
0925 07:30:55.384862 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:55.384902 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 07:30:55.385370 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:30:55.385626 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:30:55.385629 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:30:55.386183 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:30:55.386365 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:55.386369 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:55.387019 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:55.387019 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:55.387798 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:55.387800 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:55.388103 (+ 303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:55.388103 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:55.388196 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:55.388633 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:30:55.388640 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:30:55.389953 (+ 1313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:30:55.391399 (+ 1446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:30:55.391404 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:30:55.391405 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:30:55.391601 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:30:55.391755 (+ 154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:30:55.391756 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:30:55.391756 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:30:55.391844 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:30:55.392108 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:30:55.392111 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:30:55.392264 (+ 153us) client_negotiation.cc:769] Sending connection context
0925 07:30:55.392353 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 07:30:55.392491 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":78,"threads_started":1}
I20260925 07:30:55.393182 18016 heartbeater.cc:347] Connected to a master server at 127.9.66.62:32879
I20260925 07:30:55.393282 18016 heartbeater.cc:464] Registering TS with master...
I20260925 07:30:55.393441 18016 heartbeater.cc:511] Master 127.9.66.62:32879 requested a full tablet report, sending...
I20260925 07:30:55.393836 17690 ts_manager.cc:194] Registered new tserver with Master: c2ba107511fb4aa88bdebfb7dd7c1078 (127.9.66.2:35345)
I20260925 07:30:55.394197 17690 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.66.2@KRBTEST.COM'} at 127.9.66.2:51887
WARNING: no policy specified for kudu/127.9.66.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.66.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.66.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.3.keytab.
Entry for principal kudu/127.9.66.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.3.keytab.
I20260925 07:30:55.415864 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.66.3:0
--local_ip_for_outbound_sockets=127.9.66.3
--webserver_interface=127.9.66.3
--webserver_port=0
--tserver_master_addrs=127.9.66.62:32879
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.3.keytab
--principal=kudu/127.9.66.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:42819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:30:55.496953 18021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:30:55.497150 18021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:30:55.497174 18021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:30:55.498262 18021 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:30:55.498302 18021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:30:55.498330 18021 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.66.3
I20260925 07:30:55.499950 18021 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:42819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.3.keytab
--principal=kudu/127.9.66.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.66.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.66.3
--webserver_port=0
--tserver_master_addrs=127.9.66.62:32879
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18021
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.66.3
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:30:55.500165 18021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:30:55.500329 18021 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:30:55.505095 18027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:30:55.505116 18029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:30:55.505115 18026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:30:55.505347 18021 server_base.cc:1034] running on GCE node
I20260925 07:30:55.505609 18021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:30:55.505815 18021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:30:55.506996 18021 hybrid_clock.cc:648] HybridClock initialized: now 1790321455506974 us; error 35 us; skew 500 ppm
Sep 25 07:30:55 dist-test-slave-57fn krb5kdc[17646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321455, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:30:55.509308 18021 init.cc:377] Logged in from keytab as kudu/127.9.66.3@KRBTEST.COM (short username kudu)
I20260925 07:30:55.509764 18021 webserver.cc:460] Webserver started at http://127.9.66.3:38683/ using document root <none> and password file <none>
I20260925 07:30:55.509989 18021 fs_manager.cc:359] Metadata directory not provided
I20260925 07:30:55.510053 18021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:30:55.510202 18021 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:30:55.511196 18021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/minicluster-data/ts-2/data/instance:
uuid: "eefe6afa7a874af880d3e4fa94dd24b5"
format_stamp: "Formatted at 2026-09-25 07:30:55 on dist-test-slave-57fn"
server_key: "3cff8178dc73b9fd0ba6e1a87790aa2e"
server_key_iv: "7dc1b69f31f8a7a3d22602441af1734d"
server_key_version: "encryptionkey@0"
I20260925 07:30:55.511540 18021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/minicluster-data/ts-2/wal/instance:
uuid: "eefe6afa7a874af880d3e4fa94dd24b5"
format_stamp: "Formatted at 2026-09-25 07:30:55 on dist-test-slave-57fn"
server_key: "3cff8178dc73b9fd0ba6e1a87790aa2e"
server_key_iv: "7dc1b69f31f8a7a3d22602441af1734d"
server_key_version: "encryptionkey@0"
I20260925 07:30:55.512799 18021 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 07:30:55.513466 18036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:55.513629 18021 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 07:30:55.513718 18021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/minicluster-data/ts-2/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/minicluster-data/ts-2/wal
uuid: "eefe6afa7a874af880d3e4fa94dd24b5"
format_stamp: "Formatted at 2026-09-25 07:30:55 on dist-test-slave-57fn"
server_key: "3cff8178dc73b9fd0ba6e1a87790aa2e"
server_key_iv: "7dc1b69f31f8a7a3d22602441af1734d"
server_key_version: "encryptionkey@0"
I20260925 07:30:55.513796 18021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:30:55.536662 18021 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:30:55.538534 18021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:30:55.538702 18021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:30:55.538926 18021 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:30:55.539234 18021 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:30:55.539284 18021 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:55.539337 18021 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:30:55.539361 18021 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:55.545077 18021 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.3:33449
I20260925 07:30:55.545114 18149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.3:33449 every 8 connection(s)
I20260925 07:30:55.545421 18021 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/minicluster-data/ts-2/data/info.pb
I20260925 07:30:55.550170 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 18021
I20260925 07:30:55.550295 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/minicluster-data/ts-2/wal/instance
I20260925 07:30:55.550518 9480 external_mini_cluster.cc:1428] Setting key 16d5ab52f65993d7218ccb825dba8004
Sep 25 07:30:55 dist-test-slave-57fn krb5kdc[17646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321455, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.3@KRBTEST.COM for kudu/127.9.66.62@KRBTEST.COM
I20260925 07:30:55.554617 17730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:55.546131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.3:42599 (local address 127.9.66.62:32879)
0925 07:30:55.546181 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:55.546184 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:55.546396 (+ 212us) server_negotiation.cc:407] Connection header received
0925 07:30:55.546919 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:55.546920 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:55.546930 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:55.546980 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:30:55.547977 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:55.548536 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:55.549646 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:55.549839 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:55.549933 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:55.552354 (+ 2421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:30:55.552355 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:30:55.552356 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:30:55.552362 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:30:55.553485 (+ 1123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:55.553888 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:55.553888 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:55.553889 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:55.553940 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:55.554134 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:55.554134 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:55.554134 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:55.554235 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:30:55.554283 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:55.554495 (+ 212us) server_negotiation.cc:299] Negotiation successful
0925 07:30:55.554553 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 07:30:55.554838 18152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:55.546122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:32879 (local address 127.9.66.3:42599)
0925 07:30:55.546281 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:55.546316 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 07:30:55.546842 (+ 526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:30:55.547032 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:30:55.547035 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:30:55.547694 (+ 659us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:30:55.547874 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:55.547878 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:55.548631 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:55.548632 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:55.549559 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:55.549572 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:55.549941 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:55.549942 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:55.550084 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:55.550648 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:30:55.550656 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:30:55.552277 (+ 1621us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:30:55.553609 (+ 1332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:30:55.553616 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:30:55.553618 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:30:55.553827 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:30:55.553994 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:30:55.553994 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:30:55.553994 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:30:55.554083 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:30:55.554283 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:30:55.554287 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:30:55.554395 (+ 108us) client_negotiation.cc:769] Sending connection context
0925 07:30:55.554473 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 07:30:55.554588 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":87,"threads_started":1}
I20260925 07:30:55.555294 18150 heartbeater.cc:347] Connected to a master server at 127.9.66.62:32879
I20260925 07:30:55.555368 18150 heartbeater.cc:464] Registering TS with master...
I20260925 07:30:55.555545 18150 heartbeater.cc:511] Master 127.9.66.62:32879 requested a full tablet report, sending...
I20260925 07:30:55.555902 17690 ts_manager.cc:194] Registered new tserver with Master: eefe6afa7a874af880d3e4fa94dd24b5 (127.9.66.3:33449)
I20260925 07:30:55.556274 17690 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.66.3@KRBTEST.COM'} at 127.9.66.3:42599
I20260925 07:30:55.562168 9480 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.9.66.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.66.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.66.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/krb5kdc/oryx_127.9.66.1.keytab.
Entry for principal oryx/127.9.66.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/krb5kdc/oryx_127.9.66.1.keytab.
WARNING: no policy specified for oryx/127.9.66.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.66.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.66.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/krb5kdc/oryx_127.9.66.2.keytab.
Entry for principal oryx/127.9.66.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/krb5kdc/oryx_127.9.66.2.keytab.
WARNING: no policy specified for oryx/127.9.66.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.66.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.66.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/krb5kdc/oryx_127.9.66.3.keytab.
Entry for principal oryx/127.9.66.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/krb5kdc/oryx_127.9.66.3.keytab.
I20260925 07:30:55.630245 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 17753
I20260925 07:30:55.634693 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 17887
I20260925 07:30:55.638882 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 18021
I20260925 07:30:55.642972 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 17659
I20260925 07:30:55.647950 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.66.62:32879
--webserver_interface=127.9.66.62
--webserver_port=41705
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab
--principal=kudu/127.9.66.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:42819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.66.62:32879
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:30:55.727531 18159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:30:55.727715 18159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:30:55.727736 18159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:30:55.728801 18159 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:30:55.728835 18159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:30:55.728849 18159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:30:55.728863 18159 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:30:55.728874 18159 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:30:55.730229 18159 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:42819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.66.62:32879
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab
--principal=kudu/127.9.66.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.66.62:32879
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.66.62
--webserver_port=41705
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18159
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:30:55.730427 18159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:30:55.730602 18159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:30:55.735261 18167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:30:55.735291 18164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:30:55.735380 18159 server_base.cc:1034] running on GCE node
W20260925 07:30:55.735261 18165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:30:55.735680 18159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:30:55.735899 18159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:30:55.737032 18159 hybrid_clock.cc:648] HybridClock initialized: now 1790321455737021 us; error 32 us; skew 500 ppm
Sep 25 07:30:55 dist-test-slave-57fn krb5kdc[17646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321455, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:30:55.739292 18159 init.cc:377] Logged in from keytab as kudu/127.9.66.62@KRBTEST.COM (short username kudu)
I20260925 07:30:55.739696 18159 webserver.cc:460] Webserver started at http://127.9.66.62:41705/ using document root <none> and password file <none>
I20260925 07:30:55.739916 18159 fs_manager.cc:359] Metadata directory not provided
I20260925 07:30:55.739984 18159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:30:55.741219 18159 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 07:30:55.741887 18174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:55.742061 18159 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:55.742148 18159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/minicluster-data/master-0/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/minicluster-data/master-0/wal
uuid: "aaf5cbfc86744cd1956ea2aa4645a979"
format_stamp: "Formatted at 2026-09-25 07:30:55 on dist-test-slave-57fn"
server_key: "0a094162d18b77073a1e115655411fbb"
server_key_iv: "7d5963d4a2f2c5024955ff502f79b7da"
server_key_version: "encryptionkey@0"
I20260925 07:30:55.742447 18159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:30:55.767810 18159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:30:55.769680 18159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:30:55.769820 18159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:30:55.773756 18159 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.62:32879
I20260925 07:30:55.773787 18226 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.62:32879 every 8 connection(s)
I20260925 07:30:55.774158 18159 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
I20260925 07:30:55.775506 18227 sys_catalog.cc:263] Verifying existing consensus state
I20260925 07:30:55.776125 18227 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aaf5cbfc86744cd1956ea2aa4645a979: Bootstrap starting.
I20260925 07:30:55.782383 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 18159
I20260925 07:30:55.782678 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.66.1:41339
--local_ip_for_outbound_sockets=127.9.66.1
--tserver_master_addrs=127.9.66.62:32879
--webserver_port=38313
--webserver_interface=127.9.66.1
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.1.keytab
--principal=kudu/127.9.66.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:42819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.9.66.1
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/krb5kdc/oryx_127.9.66.1.keytab with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 07:30:55.783957 18227 log.cc:824] T 00000000000000000000000000000000 P aaf5cbfc86744cd1956ea2aa4645a979: Log is configured to *not* fsync() on all Append() calls
I20260925 07:30:55.785493 18227 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aaf5cbfc86744cd1956ea2aa4645a979: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 07:30:55.785820 18227 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aaf5cbfc86744cd1956ea2aa4645a979: Bootstrap complete.
I20260925 07:30:55.787976 18227 raft_consensus.cc:348] T 00000000000000000000000000000000 P aaf5cbfc86744cd1956ea2aa4645a979 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaf5cbfc86744cd1956ea2aa4645a979" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 32879 } }
I20260925 07:30:55.788203 18227 raft_consensus.cc:729] T 00000000000000000000000000000000 P aaf5cbfc86744cd1956ea2aa4645a979 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: aaf5cbfc86744cd1956ea2aa4645a979, State: Initialized, Role: FOLLOWER
I20260925 07:30:55.788380 18227 consensus_queue.cc:260] T 00000000000000000000000000000000 P aaf5cbfc86744cd1956ea2aa4645a979 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaf5cbfc86744cd1956ea2aa4645a979" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 32879 } }
I20260925 07:30:55.788448 18227 raft_consensus.cc:388] T 00000000000000000000000000000000 P aaf5cbfc86744cd1956ea2aa4645a979 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 07:30:55.788497 18227 raft_consensus.cc:482] T 00000000000000000000000000000000 P aaf5cbfc86744cd1956ea2aa4645a979 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 07:30:55.788542 18227 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aaf5cbfc86744cd1956ea2aa4645a979 [term 1 FOLLOWER]: Advancing to term 2
I20260925 07:30:55.789420 18227 raft_consensus.cc:504] T 00000000000000000000000000000000 P aaf5cbfc86744cd1956ea2aa4645a979 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaf5cbfc86744cd1956ea2aa4645a979" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 32879 } }
I20260925 07:30:55.789551 18227 leader_election.cc:302] T 00000000000000000000000000000000 P aaf5cbfc86744cd1956ea2aa4645a979 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aaf5cbfc86744cd1956ea2aa4645a979; no voters:
I20260925 07:30:55.789750 18227 leader_election.cc:288] T 00000000000000000000000000000000 P aaf5cbfc86744cd1956ea2aa4645a979 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 07:30:55.789783 18231 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aaf5cbfc86744cd1956ea2aa4645a979 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 07:30:55.789901 18231 raft_consensus.cc:686] T 00000000000000000000000000000000 P aaf5cbfc86744cd1956ea2aa4645a979 [term 2 LEADER]: Becoming Leader. State: Replica: aaf5cbfc86744cd1956ea2aa4645a979, State: Running, Role: LEADER
I20260925 07:30:55.789983 18231 consensus_queue.cc:237] T 00000000000000000000000000000000 P aaf5cbfc86744cd1956ea2aa4645a979 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaf5cbfc86744cd1956ea2aa4645a979" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 32879 } }
I20260925 07:30:55.790006 18227 sys_catalog.cc:564] T 00000000000000000000000000000000 P aaf5cbfc86744cd1956ea2aa4645a979 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:30:55.790282 18233 sys_catalog.cc:455] T 00000000000000000000000000000000 P aaf5cbfc86744cd1956ea2aa4645a979 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "aaf5cbfc86744cd1956ea2aa4645a979" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaf5cbfc86744cd1956ea2aa4645a979" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 32879 } } }
I20260925 07:30:55.790364 18233 sys_catalog.cc:458] T 00000000000000000000000000000000 P aaf5cbfc86744cd1956ea2aa4645a979 [sys.catalog]: This master's current role is: LEADER
I20260925 07:30:55.790544 18235 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 07:30:55.791003 18235 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 07:30:55.791173 18235 catalog_manager.cc:1259] Loaded cluster ID: bc747919f91e49cab589258a827efa7f
I20260925 07:30:55.791222 18235 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 07:30:55.791549 18232 sys_catalog.cc:455] T 00000000000000000000000000000000 P aaf5cbfc86744cd1956ea2aa4645a979 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aaf5cbfc86744cd1956ea2aa4645a979. Latest consensus state: current_term: 2 leader_uuid: "aaf5cbfc86744cd1956ea2aa4645a979" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aaf5cbfc86744cd1956ea2aa4645a979" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 32879 } } }
I20260925 07:30:55.791625 18232 sys_catalog.cc:458] T 00000000000000000000000000000000 P aaf5cbfc86744cd1956ea2aa4645a979 [sys.catalog]: This master's current role is: LEADER
I20260925 07:30:55.792296 18235 catalog_manager.cc:1504] Loading token signing keys...
I20260925 07:30:55.792469 18235 catalog_manager.cc:5879] T 00000000000000000000000000000000 P aaf5cbfc86744cd1956ea2aa4645a979: Loaded TSK: 0
I20260925 07:30:55.792672 18235 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 07:30:55.861846 18229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:30:55.862092 18229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:30:55.862134 18229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:30:55.863297 18229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:30:55.863360 18229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:30:55.863406 18229 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.66.1
I20260925 07:30:55.864780 18229 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:42819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/krb5kdc/oryx_127.9.66.1.keytab
--principal=oryx/127.9.66.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.66.1:41339
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.66.1
--webserver_port=38313
--tserver_master_addrs=127.9.66.62:32879
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18229
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.66.1
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:30:55.865010 18229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:30:55.865222 18229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:30:55.870965 18254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:30:55.871025 18253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:30:55.870965 18256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:30:55.871310 18229 server_base.cc:1034] running on GCE node
I20260925 07:30:55.871490 18229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:30:55.871730 18229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:30:55.872877 18229 hybrid_clock.cc:648] HybridClock initialized: now 1790321455872862 us; error 34 us; skew 500 ppm
Sep 25 07:30:55 dist-test-slave-57fn krb5kdc[17646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321455, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.66.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:30:55.875496 18229 init.cc:377] Logged in from keytab as oryx/127.9.66.1@KRBTEST.COM (short username oryx)
I20260925 07:30:55.875995 18229 webserver.cc:460] Webserver started at http://127.9.66.1:38313/ using document root <none> and password file <none>
I20260925 07:30:55.876241 18229 fs_manager.cc:359] Metadata directory not provided
I20260925 07:30:55.876312 18229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:30:55.877978 18229 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 07:30:55.878648 18263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:55.878824 18229 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 07:30:55.878913 18229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/minicluster-data/ts-0/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/minicluster-data/ts-0/wal
uuid: "f858a8705331410d9a1703be45bf92b2"
format_stamp: "Formatted at 2026-09-25 07:30:55 on dist-test-slave-57fn"
server_key: "d780e833a18f69bda16e7d506d674477"
server_key_iv: "206156eb9c70fecf4fdd53fe2b98d10d"
server_key_version: "encryptionkey@0"
I20260925 07:30:55.879330 18229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:30:55.896557 18229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:30:55.898849 18229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:30:55.899003 18229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:30:55.899268 18229 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:30:55.899747 18229 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:30:55.899798 18229 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:55.899892 18229 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:30:55.899924 18229 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:55.907080 18229 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.1:41339
I20260925 07:30:55.907163 18376 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.1:41339 every 8 connection(s)
I20260925 07:30:55.907575 18229 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/minicluster-data/ts-0/data/info.pb
Sep 25 07:30:55 dist-test-slave-57fn krb5kdc[17646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.66.1@KRBTEST.COM for oryx/127.9.66.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 07:30:55 dist-test-slave-57fn krb5kdc[17646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.66.1@KRBTEST.COM for oryx/127.9.66.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 07:30:55.916644 18379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:55.908253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:32879 (local address 127.9.66.1:44453)
0925 07:30:55.908435 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:55.908473 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 07:30:55.908917 (+ 444us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:30:55.910016 (+ 1099us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:30:55.910019 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:30:55.910558 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:30:55.910733 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:55.910737 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:55.912269 (+ 1532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:55.912269 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:55.913006 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:55.913008 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:55.913811 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:55.913812 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:55.913905 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:55.914398 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:30:55.914405 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:30:55.916509 (+ 2104us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.66.62:32879: Server oryx/127.9.66.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":103,"threads_started":1}
W20260925 07:30:55.916811 18380 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:30:55.908315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.1:44453 (local address 127.9.66.62:32879)
0925 07:30:55.909624 (+ 1309us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:55.909632 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:55.909668 (+ 36us) server_negotiation.cc:407] Connection header received
0925 07:30:55.909790 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:55.909796 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:55.909842 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:55.909967 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:30:55.910836 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:55.912176 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:55.913084 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:55.913734 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:55.913836 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:55.916651 (+ 2815us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.66.1:44453: BlockingRecv error: recv got EOF from 127.9.66.1:44453 (error 108)
Metrics: {"server-negotiator.queue_time_us":1264,"thread_start_us":102,"threads_started":1}
W20260925 07:30:55.917310 18377 heartbeater.cc:644] Failed to heartbeat to 127.9.66.62:32879 (0 consecutive failures): Not authorized: Failed to ping master at 127.9.66.62:32879: Client connection negotiation failed: client connection to 127.9.66.62:32879: Server oryx/127.9.66.62@KRBTEST.COM not found in Kerberos database
I20260925 07:30:55.917415 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 18229
I20260925 07:30:55.917737 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.66.2:35345
--local_ip_for_outbound_sockets=127.9.66.2
--tserver_master_addrs=127.9.66.62:32879
--webserver_port=35039
--webserver_interface=127.9.66.2
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.2.keytab
--principal=kudu/127.9.66.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:42819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.9.66.2
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/krb5kdc/oryx_127.9.66.2.keytab with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 07:30:55 dist-test-slave-57fn krb5kdc[17646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.66.1@KRBTEST.COM for oryx/127.9.66.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 07:30:55 dist-test-slave-57fn krb5kdc[17646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.66.1@KRBTEST.COM for oryx/127.9.66.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 07:30:55.922674 18379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:55.917625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:32879 (local address 127.9.66.1:60075)
0925 07:30:55.917678 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:55.917693 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 07:30:55.917744 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:30:55.917886 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:30:55.917887 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:30:55.918025 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:30:55.918079 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:55.918081 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:55.918953 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:55.918954 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:55.919763 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:55.919765 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:55.920161 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:55.920162 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:55.920226 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:55.920662 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:30:55.920667 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:30:55.922607 (+ 1940us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.66.62:32879: Server oryx/127.9.66.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":34}
W20260925 07:30:55.922976 18380 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:30:55.917730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.1:60075 (local address 127.9.66.62:32879)
0925 07:30:55.917765 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:55.917768 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:55.917775 (+ 7us) server_negotiation.cc:407] Connection header received
0925 07:30:55.917798 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:55.917799 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:55.917811 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:55.917876 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:30:55.918181 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:55.918870 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:55.919846 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:55.920092 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:55.920147 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:55.922923 (+ 2776us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.66.1:60075: BlockingRecv error: recv got EOF from 127.9.66.1:60075 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
Sep 25 07:30:55 dist-test-slave-57fn krb5kdc[17646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.66.1@KRBTEST.COM for oryx/127.9.66.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 07:30:55 dist-test-slave-57fn krb5kdc[17646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.66.1@KRBTEST.COM for oryx/127.9.66.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 07:30:55.928500 18379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:55.923425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:32879 (local address 127.9.66.1:46515)
0925 07:30:55.923521 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:55.923532 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 07:30:55.923588 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:30:55.923740 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:30:55.923741 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:30:55.923918 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:30:55.923952 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:55.923954 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:55.924850 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:55.924851 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:55.925711 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:55.925714 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:55.926117 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:55.926118 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:55.926180 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:55.926590 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:30:55.926595 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:30:55.928438 (+ 1843us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.66.62:32879: Server oryx/127.9.66.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":81}
W20260925 07:30:55.928619 18377 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 07:30:55.928555 18380 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:30:55.923480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.1:46515 (local address 127.9.66.62:32879)
0925 07:30:55.923514 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:55.923516 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:55.923564 (+ 48us) server_negotiation.cc:407] Connection header received
0925 07:30:55.923655 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:55.923655 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:55.923664 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:55.923727 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:30:55.924032 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:55.924767 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:55.925796 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:55.926039 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:55.926104 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:55.928499 (+ 2395us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.66.1:46515: BlockingRecv error: recv got EOF from 127.9.66.1:46515 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260925 07:30:56.002082 18381 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:30:56.002313 18381 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:30:56.002352 18381 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:30:56.003537 18381 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:30:56.003597 18381 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:30:56.003639 18381 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.66.2
I20260925 07:30:56.005040 18381 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:42819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/krb5kdc/oryx_127.9.66.2.keytab
--principal=oryx/127.9.66.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.66.2:35345
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.66.2
--webserver_port=35039
--tserver_master_addrs=127.9.66.62:32879
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18381
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.66.2
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:30:56.005293 18381 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:30:56.005522 18381 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:30:56.010293 18389 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:30:56.010351 18381 server_base.cc:1034] running on GCE node
W20260925 07:30:56.010285 18386 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:30:56.010285 18387 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:30:56.010654 18381 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:30:56.010869 18381 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:30:56.012012 18381 hybrid_clock.cc:648] HybridClock initialized: now 1790321456011998 us; error 27 us; skew 500 ppm
Sep 25 07:30:56 dist-test-slave-57fn krb5kdc[17646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321456, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.66.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:30:56.014478 18381 init.cc:377] Logged in from keytab as oryx/127.9.66.2@KRBTEST.COM (short username oryx)
I20260925 07:30:56.014832 18381 webserver.cc:460] Webserver started at http://127.9.66.2:35039/ using document root <none> and password file <none>
I20260925 07:30:56.015007 18381 fs_manager.cc:359] Metadata directory not provided
I20260925 07:30:56.015054 18381 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:30:56.016212 18381 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 07:30:56.016687 18396 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:56.016845 18381 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:56.016922 18381 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/minicluster-data/ts-1/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/minicluster-data/ts-1/wal
uuid: "c2ba107511fb4aa88bdebfb7dd7c1078"
format_stamp: "Formatted at 2026-09-25 07:30:55 on dist-test-slave-57fn"
server_key: "8a4be71a1b1f32557df81e3e52089927"
server_key_iv: "859c53402400fad32e356e910373d6c1"
server_key_version: "encryptionkey@0"
I20260925 07:30:56.017220 18381 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:30:56.026147 18381 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:30:56.027814 18381 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:30:56.027957 18381 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:30:56.028155 18381 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:30:56.028436 18381 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:30:56.028491 18381 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:56.028530 18381 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:30:56.028574 18381 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:56.034117 18381 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.2:35345
I20260925 07:30:56.034192 18509 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.2:35345 every 8 connection(s)
I20260925 07:30:56.034500 18381 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/minicluster-data/ts-1/data/info.pb
Sep 25 07:30:56 dist-test-slave-57fn krb5kdc[17646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.66.2@KRBTEST.COM for oryx/127.9.66.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 07:30:56 dist-test-slave-57fn krb5kdc[17646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.66.2@KRBTEST.COM for oryx/127.9.66.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 07:30:56.041055 18512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:56.035169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:32879 (local address 127.9.66.2:47823)
0925 07:30:56.035319 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:56.035358 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 07:30:56.035802 (+ 444us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:30:56.035989 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:30:56.035992 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:30:56.036541 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:30:56.036719 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:56.036723 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:56.037336 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:56.037336 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:56.038177 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:56.038179 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:56.038472 (+ 293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:56.038473 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:56.038560 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:56.038950 (+ 390us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:30:56.038957 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:30:56.040933 (+ 1976us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.66.62:32879: Server oryx/127.9.66.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":82,"threads_started":1}
W20260925 07:30:56.041052 18380 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:30:56.035200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.2:47823 (local address 127.9.66.62:32879)
0925 07:30:56.035254 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:56.035257 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:56.035433 (+ 176us) server_negotiation.cc:407] Connection header received
0925 07:30:56.035873 (+ 440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:56.035873 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:56.035883 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:56.035938 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:30:56.036789 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:56.037269 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:56.038243 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:56.038410 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:56.038458 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:56.041008 (+ 2550us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.66.2:47823: BlockingRecv error: recv got EOF from 127.9.66.2:47823 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
W20260925 07:30:56.041285 18510 heartbeater.cc:644] Failed to heartbeat to 127.9.66.62:32879 (0 consecutive failures): Not authorized: Failed to ping master at 127.9.66.62:32879: Client connection negotiation failed: client connection to 127.9.66.62:32879: Server oryx/127.9.66.62@KRBTEST.COM not found in Kerberos database
I20260925 07:30:56.043145 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 18381
I20260925 07:30:56.043455 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.66.3:33449
--local_ip_for_outbound_sockets=127.9.66.3
--tserver_master_addrs=127.9.66.62:32879
--webserver_port=38683
--webserver_interface=127.9.66.3
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.3.keytab
--principal=kudu/127.9.66.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:42819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.9.66.3
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/krb5kdc/oryx_127.9.66.3.keytab with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 07:30:56 dist-test-slave-57fn krb5kdc[17646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.66.2@KRBTEST.COM for oryx/127.9.66.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 07:30:56 dist-test-slave-57fn krb5kdc[17646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.66.2@KRBTEST.COM for oryx/127.9.66.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 07:30:56.045588 18512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:56.041465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:32879 (local address 127.9.66.2:35823)
0925 07:30:56.041536 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:56.041546 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 07:30:56.041581 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:30:56.041677 (+ 96us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:30:56.041678 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:30:56.041775 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:30:56.041815 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:56.041816 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:56.042336 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:56.042337 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:56.042789 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:56.042790 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:56.043089 (+ 299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:56.043091 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:56.043158 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:56.043662 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:30:56.043667 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:30:56.045491 (+ 1824us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.66.62:32879: Server oryx/127.9.66.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":27}
W20260925 07:30:56.045696 18380 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:30:56.041543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.2:35823 (local address 127.9.66.62:32879)
0925 07:30:56.041584 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:56.041585 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:56.041599 (+ 14us) server_negotiation.cc:407] Connection header received
0925 07:30:56.041616 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:56.041617 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:56.041623 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:56.041668 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:30:56.041869 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:56.042282 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:56.042848 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:56.043029 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:56.043074 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:56.045637 (+ 2563us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.66.2:35823: BlockingRecv error: recv got EOF from 127.9.66.2:35823 (error 108)
Metrics: {"server-negotiator.queue_time_us":14}
Sep 25 07:30:56 dist-test-slave-57fn krb5kdc[17646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.66.2@KRBTEST.COM for oryx/127.9.66.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 07:30:56 dist-test-slave-57fn krb5kdc[17646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.66.2@KRBTEST.COM for oryx/127.9.66.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 07:30:56.050271 18512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:56.046104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:32879 (local address 127.9.66.2:50027)
0925 07:30:56.046205 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:56.046215 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 07:30:56.046264 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:30:56.046371 (+ 107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:30:56.046371 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:30:56.046516 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:30:56.046548 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:56.046549 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:56.047114 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:56.047115 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:56.047614 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:56.047616 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:56.047945 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:56.047946 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:56.047996 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:56.048379 (+ 383us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:30:56.048384 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:30:56.050184 (+ 1800us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.66.62:32879: Server oryx/127.9.66.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":85}
W20260925 07:30:56.050287 18380 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:30:56.046164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.2:50027 (local address 127.9.66.62:32879)
0925 07:30:56.046210 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:56.046212 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:56.046260 (+ 48us) server_negotiation.cc:407] Connection header received
0925 07:30:56.046316 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:56.046317 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:56.046323 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:56.046367 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:30:56.046602 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:56.047036 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:56.047694 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:56.047866 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:56.047943 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:56.050251 (+ 2308us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.66.2:50027: BlockingRecv error: recv got EOF from 127.9.66.2:50027 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
W20260925 07:30:56.050503 18510 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 07:30:56.123234 18513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:30:56.123418 18513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:30:56.123441 18513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:30:56.124568 18513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:30:56.124608 18513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:30:56.124636 18513 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.66.3
I20260925 07:30:56.126089 18513 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:42819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/krb5kdc/oryx_127.9.66.3.keytab
--principal=oryx/127.9.66.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.66.3:33449
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.66.3
--webserver_port=38683
--tserver_master_addrs=127.9.66.62:32879
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18513
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.66.3
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:30:56.126307 18513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:30:56.126515 18513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:30:56.131328 18518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:30:56.131341 18521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:30:56.131472 18513 server_base.cc:1034] running on GCE node
W20260925 07:30:56.131340 18519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:30:56.131739 18513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:30:56.131969 18513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:30:56.133102 18513 hybrid_clock.cc:648] HybridClock initialized: now 1790321456133082 us; error 36 us; skew 500 ppm
Sep 25 07:30:56 dist-test-slave-57fn krb5kdc[17646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321456, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.66.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:30:56.135490 18513 init.cc:377] Logged in from keytab as oryx/127.9.66.3@KRBTEST.COM (short username oryx)
I20260925 07:30:56.135890 18513 webserver.cc:460] Webserver started at http://127.9.66.3:38683/ using document root <none> and password file <none>
I20260925 07:30:56.136111 18513 fs_manager.cc:359] Metadata directory not provided
I20260925 07:30:56.136185 18513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:30:56.137429 18513 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:30:56.137995 18528 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:56.138171 18513 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:56.138257 18513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/minicluster-data/ts-2/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/minicluster-data/ts-2/wal
uuid: "eefe6afa7a874af880d3e4fa94dd24b5"
format_stamp: "Formatted at 2026-09-25 07:30:55 on dist-test-slave-57fn"
server_key: "3cff8178dc73b9fd0ba6e1a87790aa2e"
server_key_iv: "7dc1b69f31f8a7a3d22602441af1734d"
server_key_version: "encryptionkey@0"
I20260925 07:30:56.138615 18513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:30:56.146751 18513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:30:56.148864 18513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:30:56.149011 18513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:30:56.149224 18513 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:30:56.149495 18513 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:30:56.149591 18513 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:56.149622 18513 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:30:56.149636 18513 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:30:56.155194 18513 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.3:33449
I20260925 07:30:56.155242 18641 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.3:33449 every 8 connection(s)
I20260925 07:30:56.155561 18513 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790321392278483-9480-0/minicluster-data/ts-2/data/info.pb
I20260925 07:30:56.158586 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 18513
Sep 25 07:30:56 dist-test-slave-57fn krb5kdc[17646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.66.3@KRBTEST.COM for oryx/127.9.66.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 07:30:56 dist-test-slave-57fn krb5kdc[17646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.66.3@KRBTEST.COM for oryx/127.9.66.62@KRBTEST.COM, Server not found in Kerberos database
W20260925 07:30:56.163048 18380 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:30:56.156179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.3:45483 (local address 127.9.66.62:32879)
0925 07:30:56.156239 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:56.156240 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:56.156426 (+ 186us) server_negotiation.cc:407] Connection header received
0925 07:30:56.156874 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:56.156875 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:56.156883 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:56.156937 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:30:56.157843 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:56.158347 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:56.159677 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:56.159901 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:56.159951 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:56.162991 (+ 3040us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.66.3:45483: BlockingRecv error: recv got EOF from 127.9.66.3:45483 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 07:30:56.163122 18644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:56.156165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:32879 (local address 127.9.66.3:45483)
0925 07:30:56.156307 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:56.156345 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 07:30:56.156796 (+ 451us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:30:56.156984 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:30:56.156987 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:30:56.157568 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:30:56.157774 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:56.157779 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:56.158434 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:56.158436 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:56.159602 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:56.159604 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:56.159976 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:56.159977 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:56.160126 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:56.160615 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:30:56.160622 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:30:56.162932 (+ 2310us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.66.62:32879: Server oryx/127.9.66.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":78,"threads_started":1}
W20260925 07:30:56.163295 18642 heartbeater.cc:644] Failed to heartbeat to 127.9.66.62:32879 (0 consecutive failures): Not authorized: Failed to ping master at 127.9.66.62:32879: Client connection negotiation failed: client connection to 127.9.66.62:32879: Server oryx/127.9.66.62@KRBTEST.COM not found in Kerberos database
I20260925 07:30:56.166584 18646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:56.159108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46948 (local address 127.9.66.62:32879)
0925 07:30:56.159216 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:56.159218 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:56.159305 (+ 87us) server_negotiation.cc:407] Connection header received
0925 07:30:56.159375 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:56.159375 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:56.159383 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:56.159441 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:30:56.160135 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:56.160724 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:56.161626 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:56.161928 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:56.161985 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:56.163235 (+ 1250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:30:56.163240 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:30:56.163240 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:30:56.163252 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:30:56.165108 (+ 1856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:56.165467 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:56.165469 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:56.165470 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:56.165600 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:30:56.165802 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:30:56.165803 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:30:56.165803 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:30:56.166107 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:30:56.166206 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 07:30:56.166378 (+ 172us) server_negotiation.cc:299] Negotiation successful
0925 07:30:56.166490 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":60,"threads_started":1}
Sep 25 07:30:56 dist-test-slave-57fn krb5kdc[17646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.66.3@KRBTEST.COM for oryx/127.9.66.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 07:30:56 dist-test-slave-57fn krb5kdc[17646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.66.3@KRBTEST.COM for oryx/127.9.66.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 07:30:56.168216 18644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:56.163482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:32879 (local address 127.9.66.3:33243)
0925 07:30:56.163526 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:56.163536 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 07:30:56.163571 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:30:56.163931 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:30:56.163932 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:30:56.164054 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:30:56.164103 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:56.164105 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:56.164952 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:56.164953 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:56.165664 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:56.165666 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:56.166070 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:56.166071 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:56.166119 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:56.166570 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:30:56.166574 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:30:56.168178 (+ 1604us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.66.62:32879: Server oryx/127.9.66.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":28}
W20260925 07:30:56.168334 18380 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:30:56.163765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.3:33243 (local address 127.9.66.62:32879)
0925 07:30:56.163817 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:56.163819 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:56.163826 (+ 7us) server_negotiation.cc:407] Connection header received
0925 07:30:56.163846 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:56.163847 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:56.163856 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:56.163921 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:30:56.164174 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:56.164877 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:56.165737 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:56.165998 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:56.166056 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:56.168264 (+ 2208us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.66.3:33243: BlockingRecv error: recv got EOF from 127.9.66.3:33243 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Sep 25 07:30:56 dist-test-slave-57fn krb5kdc[17646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.66.3@KRBTEST.COM for oryx/127.9.66.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 07:30:56 dist-test-slave-57fn krb5kdc[17646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.66.3@KRBTEST.COM for oryx/127.9.66.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 07:30:56.172127 18644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:56.168435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:32879 (local address 127.9.66.3:51731)
0925 07:30:56.168461 (+ 26us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:56.168471 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 07:30:56.168506 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:30:56.168627 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:30:56.168627 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:30:56.168720 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:30:56.168742 (+ 22us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:56.168742 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:56.169663 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:56.169664 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:56.170210 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:56.170211 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:56.170496 (+ 285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:56.170496 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:56.170528 (+ 32us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:56.170806 (+ 278us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:30:56.170809 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:30:56.172087 (+ 1278us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.66.62:32879: Server oryx/127.9.66.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":13}
W20260925 07:30:56.172235 18380 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:30:56.168461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.3:51731 (local address 127.9.66.62:32879)
0925 07:30:56.168489 (+ 28us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:56.168491 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:56.168498 (+ 7us) server_negotiation.cc:407] Connection header received
0925 07:30:56.168552 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:56.168553 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:56.168560 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:56.168613 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:30:56.168830 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:56.169543 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:56.170274 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:56.170438 (+ 164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:56.170488 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:56.172153 (+ 1665us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.66.3:51731: BlockingRecv error: recv got EOF from 127.9.66.3:51731 (error 108)
Metrics: {"server-negotiator.queue_time_us":14}
W20260925 07:30:56.172233 18642 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 25 07:30:56 dist-test-slave-57fn krb5kdc[17646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.66.1@KRBTEST.COM for oryx/127.9.66.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 07:30:56 dist-test-slave-57fn krb5kdc[17646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.66.1@KRBTEST.COM for oryx/127.9.66.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 07:30:56.933434 18647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:56.928978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:32879 (local address 127.9.66.1:59963)
0925 07:30:56.929110 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:56.929120 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 07:30:56.929167 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:30:56.929315 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:30:56.929316 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:30:56.929490 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:30:56.929578 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:56.929580 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:56.930257 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:56.930258 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:56.930798 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:56.930799 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:56.931159 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:56.931160 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:56.931222 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:56.931639 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:30:56.931644 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:30:56.933357 (+ 1713us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.66.62:32879: Server oryx/127.9.66.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":74,"threads_started":1}
W20260925 07:30:56.933629 18648 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:30:56.929058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.1:59963 (local address 127.9.66.62:32879)
0925 07:30:56.929182 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:56.929184 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:56.929219 (+ 35us) server_negotiation.cc:407] Connection header received
0925 07:30:56.929241 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:56.929241 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:56.929250 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:56.929302 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:30:56.929655 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:56.930166 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:56.930881 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:56.931078 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:56.931159 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:56.933490 (+ 2331us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.66.1:59963: BlockingRecv error: recv got EOF from 127.9.66.1:59963 (error 108)
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":79,"threads_started":1}
Sep 25 07:30:57 dist-test-slave-57fn krb5kdc[17646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.66.2@KRBTEST.COM for oryx/127.9.66.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 07:30:57 dist-test-slave-57fn krb5kdc[17646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.66.2@KRBTEST.COM for oryx/127.9.66.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 07:30:57.055655 18649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:57.051055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:32879 (local address 127.9.66.2:32779)
0925 07:30:57.051214 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:57.051226 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 07:30:57.051313 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:30:57.051447 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:30:57.051448 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:30:57.051627 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:30:57.051671 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:57.051673 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:57.052308 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:57.052309 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:57.052866 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:57.052867 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:57.053259 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:57.053260 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:57.053318 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:57.053838 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:30:57.053843 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:30:57.055591 (+ 1748us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.66.62:32879: Server oryx/127.9.66.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":85,"threads_started":1}
W20260925 07:30:57.055739 18648 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:30:57.051152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.2:32779 (local address 127.9.66.62:32879)
0925 07:30:57.051218 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:57.051221 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:57.051299 (+ 78us) server_negotiation.cc:407] Connection header received
0925 07:30:57.051365 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:57.051366 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:57.051378 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:57.051442 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:30:57.051746 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:57.052233 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:57.052943 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:57.053165 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:57.053267 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:57.055678 (+ 2411us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.66.2:32779: BlockingRecv error: recv got EOF from 127.9.66.2:32779 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 25 07:30:57 dist-test-slave-57fn krb5kdc[17646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.66.3@KRBTEST.COM for oryx/127.9.66.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 07:30:57 dist-test-slave-57fn krb5kdc[17646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.66.3@KRBTEST.COM for oryx/127.9.66.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 07:30:57.177016 18650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:57.172701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:32879 (local address 127.9.66.3:42681)
0925 07:30:57.172816 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:57.172827 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 07:30:57.172878 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:30:57.173005 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:30:57.173006 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:30:57.173147 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:30:57.173189 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:57.173191 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:57.173798 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:57.173798 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:57.174305 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:57.174306 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:57.174597 (+ 291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:57.174598 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:57.174661 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:57.175082 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:30:57.175088 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:30:57.176913 (+ 1825us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.66.62:32879: Server oryx/127.9.66.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":69,"threads_started":1}
W20260925 07:30:57.177055 18648 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:30:57.172831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.3:42681 (local address 127.9.66.62:32879)
0925 07:30:57.172903 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:57.172904 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:57.172911 (+ 7us) server_negotiation.cc:407] Connection header received
0925 07:30:57.172930 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:57.172931 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:57.172942 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:57.172991 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:30:57.173243 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:57.173738 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:57.174361 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:57.174531 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:57.174576 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:57.176990 (+ 2414us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.66.3:42681: BlockingRecv error: recv got EOF from 127.9.66.3:42681 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 25 07:30:57 dist-test-slave-57fn krb5kdc[17646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.66.1@KRBTEST.COM for oryx/127.9.66.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 07:30:57 dist-test-slave-57fn krb5kdc[17646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.66.1@KRBTEST.COM for oryx/127.9.66.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 07:30:57.938426 18651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:57.933886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:32879 (local address 127.9.66.1:52113)
0925 07:30:57.934047 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:57.934057 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 07:30:57.934122 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:30:57.934252 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:30:57.934253 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:30:57.934401 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:30:57.934447 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:57.934448 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:57.935078 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:57.935079 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:57.935612 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:57.935614 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:57.935937 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:57.935937 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:57.935978 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:57.936505 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:30:57.936510 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:30:57.938336 (+ 1826us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.66.62:32879: Server oryx/127.9.66.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":81,"threads_started":1}
W20260925 07:30:57.938469 18652 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:30:57.933961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.1:52113 (local address 127.9.66.62:32879)
0925 07:30:57.934070 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:57.934080 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:57.934115 (+ 35us) server_negotiation.cc:407] Connection header received
0925 07:30:57.934173 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:57.934173 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:57.934182 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:57.934237 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:30:57.934503 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:57.934990 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:57.935692 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:57.935877 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:57.935919 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:57.938404 (+ 2485us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.66.1:52113: BlockingRecv error: recv got EOF from 127.9.66.1:52113 (error 108)
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":64,"threads_started":1}
Sep 25 07:30:58 dist-test-slave-57fn krb5kdc[17646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.66.2@KRBTEST.COM for oryx/127.9.66.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 07:30:58 dist-test-slave-57fn krb5kdc[17646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.66.2@KRBTEST.COM for oryx/127.9.66.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 07:30:58.060272 18653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:58.056168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:32879 (local address 127.9.66.2:44185)
0925 07:30:58.056327 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:58.056336 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 07:30:58.056388 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:30:58.056506 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:30:58.056507 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:30:58.056658 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:30:58.056704 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:58.056706 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:58.057296 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:58.057297 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:58.057873 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:58.057874 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:58.058159 (+ 285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:58.058159 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:58.058197 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:58.058572 (+ 375us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:30:58.058576 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:30:58.060197 (+ 1621us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.66.62:32879: Server oryx/127.9.66.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":91,"threads_started":1}
W20260925 07:30:58.060374 18652 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:30:58.056266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.2:44185 (local address 127.9.66.62:32879)
0925 07:30:58.056331 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:58.056333 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:58.056376 (+ 43us) server_negotiation.cc:407] Connection header received
0925 07:30:58.056439 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:58.056439 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:58.056448 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:58.056487 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:30:58.056761 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:58.057239 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:58.057934 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:58.058105 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:58.058151 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:58.060284 (+ 2133us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.66.2:44185: BlockingRecv error: recv got EOF from 127.9.66.2:44185 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 25 07:30:58 dist-test-slave-57fn krb5kdc[17646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.66.3@KRBTEST.COM for oryx/127.9.66.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 07:30:58 dist-test-slave-57fn krb5kdc[17646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.66.3@KRBTEST.COM for oryx/127.9.66.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 07:30:58.181805 18654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:58.177511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:32879 (local address 127.9.66.3:45649)
0925 07:30:58.177656 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:58.177669 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 07:30:58.177728 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:30:58.177852 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:30:58.177853 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:30:58.178012 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:30:58.178071 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:58.178073 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:58.178728 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:58.178729 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:58.179279 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:58.179281 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:58.179572 (+ 291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:58.179573 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:58.179614 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:58.179999 (+ 385us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:30:58.180003 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:30:58.181743 (+ 1740us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.66.62:32879: Server oryx/127.9.66.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":100,"threads_started":1}
W20260925 07:30:58.181883 18652 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:30:58.177638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.3:45649 (local address 127.9.66.62:32879)
0925 07:30:58.177699 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:58.177700 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:58.177711 (+ 11us) server_negotiation.cc:407] Connection header received
0925 07:30:58.177778 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:58.177779 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:58.177788 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:58.177837 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:30:58.178132 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:58.178601 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:58.179348 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:58.179513 (+ 165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:58.179558 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:58.181810 (+ 2252us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.66.3:45649: BlockingRecv error: recv got EOF from 127.9.66.3:45649 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Sep 25 07:30:58 dist-test-slave-57fn krb5kdc[17646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.66.1@KRBTEST.COM for oryx/127.9.66.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 07:30:58 dist-test-slave-57fn krb5kdc[17646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.66.1@KRBTEST.COM for oryx/127.9.66.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 07:30:58.942775 18655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:58.938846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:32879 (local address 127.9.66.1:49209)
0925 07:30:58.938967 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:58.938975 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 07:30:58.939016 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:30:58.939147 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:30:58.939148 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:30:58.939297 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:30:58.939343 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:58.939344 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:58.939942 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:58.939942 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:58.940483 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:58.940486 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:58.940774 (+ 288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:58.940775 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:58.940812 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:58.941205 (+ 393us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:30:58.941209 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:30:58.942716 (+ 1507us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.66.62:32879: Server oryx/127.9.66.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":69,"threads_started":1}
W20260925 07:30:58.942853 18656 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:30:58.938902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.1:49209 (local address 127.9.66.62:32879)
0925 07:30:58.939007 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:58.939009 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:58.939018 (+ 9us) server_negotiation.cc:407] Connection header received
0925 07:30:58.939078 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:58.939078 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:58.939087 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:58.939131 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:30:58.939412 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:58.939886 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:58.940551 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:58.940721 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:58.940768 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:58.942791 (+ 2023us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.66.1:49209: BlockingRecv error: recv got EOF from 127.9.66.1:49209 (error 108)
Metrics: {"server-negotiator.queue_time_us":84,"thread_start_us":62,"threads_started":1}
Sep 25 07:30:59 dist-test-slave-57fn krb5kdc[17646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.66.2@KRBTEST.COM for oryx/127.9.66.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 07:30:59 dist-test-slave-57fn krb5kdc[17646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.66.2@KRBTEST.COM for oryx/127.9.66.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 07:30:59.064754 18657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:59.060695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:32879 (local address 127.9.66.2:46005)
0925 07:30:59.060840 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:59.060850 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 07:30:59.060893 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:30:59.061032 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:30:59.061032 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:30:59.061187 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:30:59.061246 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:59.061248 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:59.061901 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:59.061902 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:59.062403 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:59.062404 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:59.062689 (+ 285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:59.062690 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:59.062728 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:59.063146 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:30:59.063150 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:30:59.064700 (+ 1550us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.66.62:32879: Server oryx/127.9.66.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":91,"threads_started":1}
W20260925 07:30:59.064872 18656 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:30:59.060870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.2:46005 (local address 127.9.66.62:32879)
0925 07:30:59.060915 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:59.060917 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:59.060925 (+ 8us) server_negotiation.cc:407] Connection header received
0925 07:30:59.060948 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:59.060948 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:59.060964 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:59.061017 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:30:59.061306 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:59.061837 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:59.062462 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:59.062635 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:59.062680 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:59.064796 (+ 2116us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.66.2:46005: BlockingRecv error: recv got EOF from 127.9.66.2:46005 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Sep 25 07:30:59 dist-test-slave-57fn krb5kdc[17646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.66.3@KRBTEST.COM for oryx/127.9.66.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 07:30:59 dist-test-slave-57fn krb5kdc[17646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.66.3@KRBTEST.COM for oryx/127.9.66.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 07:30:59.187803 18658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:59.182284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:32879 (local address 127.9.66.3:37531)
0925 07:30:59.182435 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:59.182447 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 07:30:59.182522 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:30:59.182666 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:30:59.182667 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:30:59.182895 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:30:59.182951 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:59.182953 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:59.183771 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:59.183772 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:59.184767 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:59.184770 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:59.185100 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:59.185101 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:59.185171 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:59.185698 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:30:59.185704 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:30:59.187704 (+ 2000us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.66.62:32879: Server oryx/127.9.66.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":91,"threads_started":1}
W20260925 07:30:59.187822 18656 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:30:59.182370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.3:37531 (local address 127.9.66.62:32879)
0925 07:30:59.182426 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:59.182429 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:59.182512 (+ 83us) server_negotiation.cc:407] Connection header received
0925 07:30:59.182586 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:59.182587 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:59.182597 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:59.182644 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:30:59.183021 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:59.183704 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:59.184857 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:59.185034 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:59.185086 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:59.187775 (+ 2689us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.66.3:37531: BlockingRecv error: recv got EOF from 127.9.66.3:37531 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
Sep 25 07:30:59 dist-test-slave-57fn krb5kdc[17646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.66.1@KRBTEST.COM for oryx/127.9.66.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 07:30:59 dist-test-slave-57fn krb5kdc[17646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.66.1@KRBTEST.COM for oryx/127.9.66.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 07:30:59.947331 18659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:30:59.943177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:32879 (local address 127.9.66.1:48147)
0925 07:30:59.943317 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 07:30:59.943326 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 07:30:59.943379 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:30:59.943509 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:30:59.943510 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:30:59.943658 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:30:59.943703 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:59.943705 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:59.944272 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:59.944272 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:59.944750 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:30:59.944751 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:30:59.945059 (+ 308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:30:59.945060 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:30:59.945094 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:59.945580 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:30:59.945584 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:30:59.947277 (+ 1693us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.66.62:32879: Server oryx/127.9.66.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":93,"threads_started":1}
W20260925 07:30:59.947440 18660 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:30:59.943241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.1:48147 (local address 127.9.66.62:32879)
0925 07:30:59.943332 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0925 07:30:59.943334 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:30:59.943364 (+ 30us) server_negotiation.cc:407] Connection header received
0925 07:30:59.943432 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:30:59.943432 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:30:59.943441 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:30:59.943497 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:30:59.943759 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:59.944218 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:59.944815 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:30:59.945002 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:30:59.945049 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:30:59.947363 (+ 2314us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.66.1:48147: BlockingRecv error: recv got EOF from 127.9.66.1:48147 (error 108)
Metrics: {"server-negotiator.queue_time_us":74,"thread_start_us":46,"threads_started":1}
Sep 25 07:31:00 dist-test-slave-57fn krb5kdc[17646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.66.2@KRBTEST.COM for oryx/127.9.66.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 07:31:00 dist-test-slave-57fn krb5kdc[17646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.66.2@KRBTEST.COM for oryx/127.9.66.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 07:31:00.070057 18661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:31:00.065340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:32879 (local address 127.9.66.2:38619)
0925 07:31:00.065623 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0925 07:31:00.065634 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 07:31:00.065706 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:31:00.065871 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:31:00.065872 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:31:00.066072 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:31:00.066119 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:31:00.066121 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:31:00.066812 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:31:00.066813 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:31:00.067354 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:31:00.067355 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:31:00.067722 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:31:00.067723 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:31:00.067778 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:31:00.068262 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:31:00.068267 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:31:00.069963 (+ 1696us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.66.62:32879: Server oryx/127.9.66.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":89,"threads_started":1}
W20260925 07:31:00.070106 18660 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:31:00.065425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.2:38619 (local address 127.9.66.62:32879)
0925 07:31:00.065606 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 07:31:00.065608 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:31:00.065700 (+ 92us) server_negotiation.cc:407] Connection header received
0925 07:31:00.065772 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:31:00.065773 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:31:00.065791 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:31:00.065863 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:31:00.066206 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:31:00.066729 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:31:00.067436 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:31:00.067628 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:31:00.067723 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:31:00.070041 (+ 2318us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.66.2:38619: BlockingRecv error: recv got EOF from 127.9.66.2:38619 (error 108)
Metrics: {"server-negotiator.queue_time_us":153}
Sep 25 07:31:00 dist-test-slave-57fn krb5kdc[17646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.66.3@KRBTEST.COM for oryx/127.9.66.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 07:31:00 dist-test-slave-57fn krb5kdc[17646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.66.3@KRBTEST.COM for oryx/127.9.66.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 07:31:00.192946 18662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:31:00.188339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:32879 (local address 127.9.66.3:57963)
0925 07:31:00.188484 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 07:31:00.188500 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 07:31:00.188570 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:31:00.188692 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:31:00.188692 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:31:00.188851 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:31:00.188898 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:31:00.188899 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:31:00.189572 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:31:00.189573 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:31:00.190152 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:31:00.190154 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:31:00.190522 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:31:00.190523 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:31:00.190606 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:31:00.191091 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:31:00.191096 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:31:00.192857 (+ 1761us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.66.62:32879: Server oryx/127.9.66.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":75,"threads_started":1}
W20260925 07:31:00.192993 18660 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:31:00.188380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.3:57963 (local address 127.9.66.62:32879)
0925 07:31:00.188437 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 07:31:00.188440 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:31:00.188588 (+ 148us) server_negotiation.cc:407] Connection header received
0925 07:31:00.188628 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:31:00.188629 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:31:00.188640 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:31:00.188687 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:31:00.188950 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:31:00.189433 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:31:00.190241 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:31:00.190441 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:31:00.190520 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:31:00.192929 (+ 2409us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.66.3:57963: BlockingRecv error: recv got EOF from 127.9.66.3:57963 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Sep 25 07:31:00 dist-test-slave-57fn krb5kdc[17646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.66.1@KRBTEST.COM for oryx/127.9.66.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 07:31:00 dist-test-slave-57fn krb5kdc[17646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.66.1@KRBTEST.COM for oryx/127.9.66.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 07:31:00.952425 18663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:31:00.947790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:32879 (local address 127.9.66.1:41097)
0925 07:31:00.947937 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 07:31:00.947947 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 07:31:00.948008 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:31:00.948158 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:31:00.948159 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:31:00.948312 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:31:00.948391 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:31:00.948392 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:31:00.949068 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:31:00.949069 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:31:00.949675 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:31:00.949677 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:31:00.949980 (+ 303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:31:00.949981 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:31:00.950016 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:31:00.950451 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:31:00.950456 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:31:00.952369 (+ 1913us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.66.62:32879: Server oryx/127.9.66.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":85,"threads_started":1}
W20260925 07:31:00.952518 18664 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:31:00.947871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.1:41097 (local address 127.9.66.62:32879)
0925 07:31:00.947963 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0925 07:31:00.947967 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 07:31:00.947997 (+ 30us) server_negotiation.cc:407] Connection header received
0925 07:31:00.948074 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:31:00.948074 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:31:00.948084 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:31:00.948142 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:31:00.948508 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:31:00.949001 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:31:00.949743 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:31:00.949919 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:31:00.949964 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:31:00.952446 (+ 2482us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.66.1:41097: BlockingRecv error: recv got EOF from 127.9.66.1:41097 (error 108)
Metrics: {"server-negotiator.queue_time_us":72,"thread_start_us":42,"threads_started":1}
Sep 25 07:31:01 dist-test-slave-57fn krb5kdc[17646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.66.2@KRBTEST.COM for oryx/127.9.66.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 07:31:01 dist-test-slave-57fn krb5kdc[17646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.66.2@KRBTEST.COM for oryx/127.9.66.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 07:31:01.074777 18665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:31:01.070541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:32879 (local address 127.9.66.2:57481)
0925 07:31:01.070663 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0925 07:31:01.070674 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 07:31:01.070736 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:31:01.070854 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:31:01.070855 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:31:01.071007 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:31:01.071054 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:31:01.071056 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:31:01.071752 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:31:01.071752 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:31:01.072234 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:31:01.072235 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:31:01.072511 (+ 276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:31:01.072511 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:31:01.072544 (+ 33us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:31:01.072975 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:31:01.072979 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:31:01.074717 (+ 1738us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.66.62:32879: Server oryx/127.9.66.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":73,"threads_started":1}
W20260925 07:31:01.074900 18664 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:31:01.070634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.2:57481 (local address 127.9.66.62:32879)
0925 07:31:01.070689 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 07:31:01.070691 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:31:01.070721 (+ 30us) server_negotiation.cc:407] Connection header received
0925 07:31:01.070779 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:31:01.070780 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:31:01.070789 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:31:01.070833 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:31:01.071112 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:31:01.071690 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:31:01.072295 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:31:01.072457 (+ 162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:31:01.072500 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:31:01.074802 (+ 2302us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.66.2:57481: BlockingRecv error: recv got EOF from 127.9.66.2:57481 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Sep 25 07:31:01 dist-test-slave-57fn krb5kdc[17646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.66.3@KRBTEST.COM for oryx/127.9.66.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 07:31:01 dist-test-slave-57fn krb5kdc[17646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.66.3@KRBTEST.COM for oryx/127.9.66.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 07:31:01.199072 18666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:31:01.193369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:32879 (local address 127.9.66.3:42281)
0925 07:31:01.193561 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0925 07:31:01.193578 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 07:31:01.193655 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:31:01.193815 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:31:01.193816 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:31:01.194001 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:31:01.194048 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:31:01.194049 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:31:01.194944 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:31:01.194945 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:31:01.195684 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:31:01.195687 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:31:01.196169 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:31:01.196170 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:31:01.196237 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:31:01.196661 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:31:01.196667 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:31:01.198958 (+ 2291us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.66.62:32879: Server oryx/127.9.66.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":97,"threads_started":1}
W20260925 07:31:01.199141 18664 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:31:01.193449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.3:42281 (local address 127.9.66.62:32879)
0925 07:31:01.193543 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0925 07:31:01.193545 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:31:01.193640 (+ 95us) server_negotiation.cc:407] Connection header received
0925 07:31:01.193732 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:31:01.193733 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:31:01.193758 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:31:01.193810 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:31:01.194153 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:31:01.194857 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:31:01.195760 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:31:01.196081 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:31:01.196157 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:31:01.199069 (+ 2912us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.66.3:42281: BlockingRecv error: recv got EOF from 127.9.66.3:42281 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 25 07:31:01 dist-test-slave-57fn krb5kdc[17646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.66.1@KRBTEST.COM for oryx/127.9.66.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 07:31:01 dist-test-slave-57fn krb5kdc[17646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.66.1@KRBTEST.COM for oryx/127.9.66.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 07:31:01.957237 18667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:31:01.952870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:32879 (local address 127.9.66.1:51999)
0925 07:31:01.953026 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 07:31:01.953036 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 07:31:01.953090 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:31:01.953233 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:31:01.953233 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:31:01.953407 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:31:01.953456 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:31:01.953457 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:31:01.954115 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:31:01.954116 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:31:01.954605 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:31:01.954606 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:31:01.954897 (+ 291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:31:01.954897 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:31:01.954933 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:31:01.955358 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:31:01.955362 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:31:01.957142 (+ 1780us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.66.62:32879: Server oryx/127.9.66.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":80,"threads_started":1}
W20260925 07:31:01.957291 18668 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:31:01.952928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.1:51999 (local address 127.9.66.62:32879)
0925 07:31:01.953068 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0925 07:31:01.953070 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:31:01.953078 (+ 8us) server_negotiation.cc:407] Connection header received
0925 07:31:01.953148 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:31:01.953149 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:31:01.953158 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:31:01.953213 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:31:01.953556 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:31:01.954053 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:31:01.954666 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:31:01.954840 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:31:01.954884 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:31:01.957218 (+ 2334us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.66.1:51999: BlockingRecv error: recv got EOF from 127.9.66.1:51999 (error 108)
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":92,"threads_started":1}
Sep 25 07:31:02 dist-test-slave-57fn krb5kdc[17646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.66.2@KRBTEST.COM for oryx/127.9.66.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 07:31:02 dist-test-slave-57fn krb5kdc[17646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.66.2@KRBTEST.COM for oryx/127.9.66.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 07:31:02.079587 18669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:31:02.075194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:32879 (local address 127.9.66.2:59875)
0925 07:31:02.075327 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0925 07:31:02.075337 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 07:31:02.075389 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:31:02.075518 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:31:02.075519 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:31:02.075665 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:31:02.075713 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:31:02.075714 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:31:02.076335 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:31:02.076336 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:31:02.076948 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:31:02.076950 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:31:02.077266 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:31:02.077267 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:31:02.077306 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:31:02.077770 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:31:02.077775 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:31:02.079519 (+ 1744us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.66.62:32879: Server oryx/127.9.66.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":78,"threads_started":1}
W20260925 07:31:02.079656 18668 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:31:02.075272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.2:59875 (local address 127.9.66.62:32879)
0925 07:31:02.075322 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 07:31:02.075324 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:31:02.075376 (+ 52us) server_negotiation.cc:407] Connection header received
0925 07:31:02.075443 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:31:02.075444 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:31:02.075453 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:31:02.075492 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:31:02.075772 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:31:02.076234 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:31:02.077026 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:31:02.077202 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:31:02.077248 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:31:02.079610 (+ 2362us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.66.2:59875: BlockingRecv error: recv got EOF from 127.9.66.2:59875 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Sep 25 07:31:02 dist-test-slave-57fn krb5kdc[17646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.66.3@KRBTEST.COM for oryx/127.9.66.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 07:31:02 dist-test-slave-57fn krb5kdc[17646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.66.3@KRBTEST.COM for oryx/127.9.66.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 07:31:02.203696 18670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:31:02.199606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:32879 (local address 127.9.66.3:57437)
0925 07:31:02.199747 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 07:31:02.199758 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 07:31:02.199819 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:31:02.199932 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:31:02.199932 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:31:02.200102 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:31:02.200148 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:31:02.200150 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:31:02.200745 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:31:02.200746 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:31:02.201245 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:31:02.201246 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:31:02.201556 (+ 310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:31:02.201557 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:31:02.201596 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:31:02.202008 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:31:02.202011 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:31:02.203643 (+ 1632us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.66.62:32879: Server oryx/127.9.66.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":92,"threads_started":1}
W20260925 07:31:02.203799 18668 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:31:02.199656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.3:57437 (local address 127.9.66.62:32879)
0925 07:31:02.199718 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 07:31:02.199720 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:31:02.199822 (+ 102us) server_negotiation.cc:407] Connection header received
0925 07:31:02.199854 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:31:02.199855 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:31:02.199864 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:31:02.199911 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:31:02.200219 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:31:02.200685 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:31:02.201310 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:31:02.201473 (+ 163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:31:02.201559 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:31:02.203728 (+ 2169us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.66.3:57437: BlockingRecv error: recv got EOF from 127.9.66.3:57437 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 25 07:31:02 dist-test-slave-57fn krb5kdc[17646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.66.1@KRBTEST.COM for oryx/127.9.66.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 07:31:02 dist-test-slave-57fn krb5kdc[17646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.66.1@KRBTEST.COM for oryx/127.9.66.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 07:31:02.962080 18671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:31:02.957745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:32879 (local address 127.9.66.1:55203)
0925 07:31:02.957894 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 07:31:02.957910 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 07:31:02.957967 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:31:02.958090 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:31:02.958091 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:31:02.958281 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:31:02.958327 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:31:02.958328 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:31:02.958989 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:31:02.958990 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:31:02.959497 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:31:02.959499 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:31:02.959798 (+ 299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:31:02.959799 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:31:02.959831 (+ 32us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:31:02.960221 (+ 390us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:31:02.960225 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:31:02.962026 (+ 1801us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.66.62:32879: Server oryx/127.9.66.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":84,"threads_started":1}
W20260925 07:31:02.962198 18672 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:31:02.957817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.1:55203 (local address 127.9.66.62:32879)
0925 07:31:02.957949 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 07:31:02.957964 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 07:31:02.957971 (+ 7us) server_negotiation.cc:407] Connection header received
0925 07:31:02.958018 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:31:02.958019 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:31:02.958027 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:31:02.958071 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:31:02.958405 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:31:02.958929 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:31:02.959560 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:31:02.959729 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:31:02.959786 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:31:02.962122 (+ 2336us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.66.1:55203: BlockingRecv error: recv got EOF from 127.9.66.1:55203 (error 108)
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":84,"threads_started":1}
Sep 25 07:31:03 dist-test-slave-57fn krb5kdc[17646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.66.2@KRBTEST.COM for oryx/127.9.66.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 07:31:03 dist-test-slave-57fn krb5kdc[17646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.66.2@KRBTEST.COM for oryx/127.9.66.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 07:31:03.084189 18673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:31:03.080104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:32879 (local address 127.9.66.2:49299)
0925 07:31:03.080229 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0925 07:31:03.080239 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 07:31:03.080290 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:31:03.080397 (+ 107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:31:03.080397 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:31:03.080543 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:31:03.080588 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:31:03.080590 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:31:03.081169 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:31:03.081169 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:31:03.081718 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:31:03.081720 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:31:03.082011 (+ 291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:31:03.082012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:31:03.082046 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:31:03.082446 (+ 400us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:31:03.082450 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:31:03.084137 (+ 1687us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.66.62:32879: Server oryx/127.9.66.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":74,"threads_started":1}
W20260925 07:31:03.084268 18672 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:31:03.080142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.2:49299 (local address 127.9.66.62:32879)
0925 07:31:03.080205 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 07:31:03.080207 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:31:03.080290 (+ 83us) server_negotiation.cc:407] Connection header received
0925 07:31:03.080344 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:31:03.080345 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:31:03.080353 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:31:03.080388 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:31:03.080643 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:31:03.081108 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:31:03.081780 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:31:03.081953 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:31:03.082002 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:31:03.084206 (+ 2204us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.66.2:49299: BlockingRecv error: recv got EOF from 127.9.66.2:49299 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 25 07:31:03 dist-test-slave-57fn krb5kdc[17646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.66.3@KRBTEST.COM for oryx/127.9.66.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 07:31:03 dist-test-slave-57fn krb5kdc[17646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.66.3@KRBTEST.COM for oryx/127.9.66.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 07:31:03.208631 18674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:31:03.204326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:32879 (local address 127.9.66.3:47605)
0925 07:31:03.204489 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 07:31:03.204500 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 07:31:03.204559 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:31:03.204671 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:31:03.204672 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:31:03.204814 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:31:03.204858 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:31:03.204860 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:31:03.205437 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:31:03.205438 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:31:03.206138 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:31:03.206140 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:31:03.206419 (+ 279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:31:03.206420 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:31:03.206453 (+ 33us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:31:03.206875 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:31:03.206880 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:31:03.208578 (+ 1698us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.66.62:32879: Server oryx/127.9.66.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":99,"threads_started":1}
W20260925 07:31:03.208712 18672 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:31:03.204374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.3:47605 (local address 127.9.66.62:32879)
0925 07:31:03.204432 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 07:31:03.204434 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:31:03.204557 (+ 123us) server_negotiation.cc:407] Connection header received
0925 07:31:03.204598 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:31:03.204599 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:31:03.204610 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:31:03.204653 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:31:03.204909 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:31:03.205358 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:31:03.206201 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:31:03.206367 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:31:03.206410 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:31:03.208654 (+ 2244us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.66.3:47605: BlockingRecv error: recv got EOF from 127.9.66.3:47605 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Sep 25 07:31:03 dist-test-slave-57fn krb5kdc[17646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.66.1@KRBTEST.COM for oryx/127.9.66.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 07:31:03 dist-test-slave-57fn krb5kdc[17646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.66.1@KRBTEST.COM for oryx/127.9.66.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 07:31:03.971436 18675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:31:03.962531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:32879 (local address 127.9.66.1:60141)
0925 07:31:03.962658 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0925 07:31:03.962667 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 07:31:03.962711 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:31:03.962907 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:31:03.962908 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:31:03.963058 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:31:03.963102 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:31:03.963103 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:31:03.963693 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:31:03.963694 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:31:03.964180 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:31:03.964181 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:31:03.964470 (+ 289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:31:03.964470 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:31:03.964506 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:31:03.969480 (+ 4974us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:31:03.969484 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:31:03.971376 (+ 1892us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.66.62:32879: Server oryx/127.9.66.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":70,"threads_started":1}
W20260925 07:31:03.971577 18676 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:31:03.962669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.1:60141 (local address 127.9.66.62:32879)
0925 07:31:03.962789 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0925 07:31:03.962791 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:31:03.962798 (+ 7us) server_negotiation.cc:407] Connection header received
0925 07:31:03.962819 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:31:03.962820 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:31:03.962836 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:31:03.962892 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:31:03.963160 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:31:03.963634 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:31:03.964242 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:31:03.964414 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:31:03.964458 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:31:03.971487 (+ 7029us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.66.1:60141: BlockingRecv error: recv got EOF from 127.9.66.1:60141 (error 108)
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":76,"threads_started":1}
Sep 25 07:31:04 dist-test-slave-57fn krb5kdc[17646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.66.2@KRBTEST.COM for oryx/127.9.66.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 07:31:04 dist-test-slave-57fn krb5kdc[17646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.66.2@KRBTEST.COM for oryx/127.9.66.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 07:31:04.089391 18677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:31:04.084681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:32879 (local address 127.9.66.2:38471)
0925 07:31:04.084825 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 07:31:04.084836 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 07:31:04.084894 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:31:04.085048 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:31:04.085049 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:31:04.085204 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:31:04.085250 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:31:04.085251 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:31:04.086068 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:31:04.086069 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:31:04.086590 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:31:04.086602 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:31:04.086899 (+ 297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:31:04.086899 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:31:04.086936 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:31:04.087427 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:31:04.087431 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:31:04.089300 (+ 1869us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.66.62:32879: Server oryx/127.9.66.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":79,"threads_started":1}
W20260925 07:31:04.089429 18676 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:31:04.084775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.2:38471 (local address 127.9.66.62:32879)
0925 07:31:04.084836 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 07:31:04.084838 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:31:04.084893 (+ 55us) server_negotiation.cc:407] Connection header received
0925 07:31:04.084972 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:31:04.084973 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:31:04.084982 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:31:04.085028 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:31:04.085307 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:31:04.086005 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:31:04.086664 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:31:04.086841 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:31:04.086887 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:31:04.089370 (+ 2483us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.66.2:38471: BlockingRecv error: recv got EOF from 127.9.66.2:38471 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 25 07:31:04 dist-test-slave-57fn krb5kdc[17646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.66.3@KRBTEST.COM for oryx/127.9.66.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 07:31:04 dist-test-slave-57fn krb5kdc[17646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.66.3@KRBTEST.COM for oryx/127.9.66.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 07:31:04.213358 18678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:31:04.209122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:32879 (local address 127.9.66.3:60471)
0925 07:31:04.209240 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0925 07:31:04.209250 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 07:31:04.209304 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:31:04.209415 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:31:04.209416 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:31:04.209589 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:31:04.209642 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:31:04.209643 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:31:04.210252 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:31:04.210253 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:31:04.210768 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:31:04.210769 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:31:04.211060 (+ 291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:31:04.211060 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:31:04.211099 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:31:04.211599 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:31:04.211604 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:31:04.213302 (+ 1698us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.66.62:32879: Server oryx/127.9.66.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":56,"threads_started":1}
W20260925 07:31:04.213454 18676 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:31:04.209189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.3:60471 (local address 127.9.66.62:32879)
0925 07:31:04.209266 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0925 07:31:04.209268 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:31:04.209302 (+ 34us) server_negotiation.cc:407] Connection header received
0925 07:31:04.209356 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:31:04.209356 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:31:04.209366 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:31:04.209411 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:31:04.209701 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:31:04.210189 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:31:04.210830 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:31:04.211000 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:31:04.211056 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:31:04.213387 (+ 2331us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.66.3:60471: BlockingRecv error: recv got EOF from 127.9.66.3:60471 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 25 07:31:04 dist-test-slave-57fn krb5kdc[17646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.66.1@KRBTEST.COM for oryx/127.9.66.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 07:31:04 dist-test-slave-57fn krb5kdc[17646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.66.1@KRBTEST.COM for oryx/127.9.66.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 07:31:04.976605 18679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:31:04.972065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:32879 (local address 127.9.66.1:49535)
0925 07:31:04.972213 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0925 07:31:04.972224 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 07:31:04.972290 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:31:04.972437 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:31:04.972438 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:31:04.972592 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:31:04.972641 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:31:04.972642 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:31:04.973250 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:31:04.973251 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:31:04.974079 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:31:04.974083 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:31:04.974403 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:31:04.974404 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:31:04.974440 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:31:04.974887 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:31:04.974891 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:31:04.976543 (+ 1652us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.66.62:32879: Server oryx/127.9.66.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":93,"threads_started":1}
W20260925 07:31:04.976717 18680 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:31:04.972141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.1:49535 (local address 127.9.66.62:32879)
0925 07:31:04.972280 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 07:31:04.972282 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:31:04.972290 (+ 8us) server_negotiation.cc:407] Connection header received
0925 07:31:04.972355 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:31:04.972355 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:31:04.972364 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:31:04.972414 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:31:04.972701 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:31:04.973189 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:31:04.974148 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:31:04.974348 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:31:04.974393 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:31:04.976639 (+ 2246us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.66.1:49535: BlockingRecv error: recv got EOF from 127.9.66.1:49535 (error 108)
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":76,"threads_started":1}
Sep 25 07:31:05 dist-test-slave-57fn krb5kdc[17646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.66.2@KRBTEST.COM for oryx/127.9.66.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 07:31:05 dist-test-slave-57fn krb5kdc[17646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.66.2@KRBTEST.COM for oryx/127.9.66.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 07:31:05.094448 18681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:31:05.089854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:32879 (local address 127.9.66.2:54951)
0925 07:31:05.090003 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 07:31:05.090022 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 07:31:05.090081 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:31:05.090214 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:31:05.090215 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:31:05.090440 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:31:05.090487 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:31:05.090489 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:31:05.091326 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:31:05.091327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:31:05.091899 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:31:05.091901 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:31:05.092195 (+ 294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:31:05.092196 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:31:05.092232 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:31:05.092626 (+ 394us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:31:05.092630 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:31:05.094391 (+ 1761us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.66.62:32879: Server oryx/127.9.66.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":65,"threads_started":1}
W20260925 07:31:05.094512 18680 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:31:05.089918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.2:54951 (local address 127.9.66.62:32879)
0925 07:31:05.090026 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0925 07:31:05.090032 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 07:31:05.090065 (+ 33us) server_negotiation.cc:407] Connection header received
0925 07:31:05.090135 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:31:05.090136 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:31:05.090146 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:31:05.090198 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:31:05.090554 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:31:05.091252 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:31:05.091956 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:31:05.092132 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:31:05.092182 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:31:05.094467 (+ 2285us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.66.2:54951: BlockingRecv error: recv got EOF from 127.9.66.2:54951 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Sep 25 07:31:05 dist-test-slave-57fn krb5kdc[17646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.66.3@KRBTEST.COM for oryx/127.9.66.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 07:31:05 dist-test-slave-57fn krb5kdc[17646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.66.3@KRBTEST.COM for oryx/127.9.66.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 07:31:05.218037 18682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:31:05.213882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:32879 (local address 127.9.66.3:60719)
0925 07:31:05.214031 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 07:31:05.214041 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 07:31:05.214099 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:31:05.214234 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:31:05.214234 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:31:05.214387 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:31:05.214433 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:31:05.214435 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:31:05.215027 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:31:05.215028 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:31:05.215520 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:31:05.215521 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:31:05.215827 (+ 306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:31:05.215828 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:31:05.215888 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:31:05.216300 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:31:05.216304 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:31:05.217982 (+ 1678us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.66.62:32879: Server oryx/127.9.66.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":81,"threads_started":1}
W20260925 07:31:05.218139 18680 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:31:05.213939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.3:60719 (local address 127.9.66.62:32879)
0925 07:31:05.213988 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 07:31:05.213990 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:31:05.214102 (+ 112us) server_negotiation.cc:407] Connection header received
0925 07:31:05.214151 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:31:05.214152 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:31:05.214164 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:31:05.214215 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:31:05.214498 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:31:05.214969 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:31:05.215595 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:31:05.215762 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:31:05.215808 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:31:05.218068 (+ 2260us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.66.3:60719: BlockingRecv error: recv got EOF from 127.9.66.3:60719 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Sep 25 07:31:05 dist-test-slave-57fn krb5kdc[17646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.66.1@KRBTEST.COM for oryx/127.9.66.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 07:31:05 dist-test-slave-57fn krb5kdc[17646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.66.1@KRBTEST.COM for oryx/127.9.66.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 07:31:05.981441 18683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:31:05.977148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:32879 (local address 127.9.66.1:39061)
0925 07:31:05.977312 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 07:31:05.977322 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 07:31:05.977378 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:31:05.977490 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:31:05.977490 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:31:05.977699 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:31:05.977747 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:31:05.977749 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:31:05.978366 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:31:05.978367 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:31:05.978881 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:31:05.978882 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:31:05.979183 (+ 301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:31:05.979184 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:31:05.979222 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:31:05.979632 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:31:05.979636 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:31:05.981350 (+ 1714us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.66.62:32879: Server oryx/127.9.66.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":82,"threads_started":1}
W20260925 07:31:05.981492 18684 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:31:05.977200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.1:39061 (local address 127.9.66.62:32879)
0925 07:31:05.977320 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0925 07:31:05.977322 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:31:05.977363 (+ 41us) server_negotiation.cc:407] Connection header received
0925 07:31:05.977428 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:31:05.977429 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:31:05.977438 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:31:05.977479 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:31:05.977808 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:31:05.978305 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:31:05.978942 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:31:05.979127 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:31:05.979173 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:31:05.981417 (+ 2244us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.66.1:39061: BlockingRecv error: recv got EOF from 127.9.66.1:39061 (error 108)
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":76,"threads_started":1}
Sep 25 07:31:06 dist-test-slave-57fn krb5kdc[17646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.66.2@KRBTEST.COM for oryx/127.9.66.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 07:31:06 dist-test-slave-57fn krb5kdc[17646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.66.2@KRBTEST.COM for oryx/127.9.66.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 07:31:06.099331 18685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:31:06.094890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:32879 (local address 127.9.66.2:56739)
0925 07:31:06.095052 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 07:31:06.095062 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 07:31:06.095149 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:31:06.095301 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:31:06.095302 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:31:06.095473 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:31:06.095517 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:31:06.095519 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:31:06.096153 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:31:06.096154 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:31:06.096657 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:31:06.096659 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:31:06.097041 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:31:06.097042 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:31:06.097113 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:31:06.097601 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:31:06.097605 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:31:06.099275 (+ 1670us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.66.62:32879: Server oryx/127.9.66.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":82,"threads_started":1}
W20260925 07:31:06.099438 18684 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:31:06.094968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.2:56739 (local address 127.9.66.62:32879)
0925 07:31:06.095033 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0925 07:31:06.095035 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:31:06.095131 (+ 96us) server_negotiation.cc:407] Connection header received
0925 07:31:06.095208 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:31:06.095209 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:31:06.095222 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:31:06.095293 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:31:06.095604 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:31:06.096077 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:31:06.096735 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:31:06.096955 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:31:06.097041 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:31:06.099367 (+ 2326us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.66.2:56739: BlockingRecv error: recv got EOF from 127.9.66.2:56739 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 25 07:31:06 dist-test-slave-57fn krb5kdc[17646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.66.3@KRBTEST.COM for oryx/127.9.66.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 07:31:06 dist-test-slave-57fn krb5kdc[17646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.66.3@KRBTEST.COM for oryx/127.9.66.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 07:31:06.223352 18686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:31:06.218584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:32879 (local address 127.9.66.3:43343)
0925 07:31:06.218721 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 07:31:06.218732 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 07:31:06.218798 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:31:06.218938 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:31:06.218939 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:31:06.219091 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:31:06.219134 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:31:06.219135 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:31:06.219846 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:31:06.219847 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:31:06.220356 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:31:06.220357 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:31:06.220748 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:31:06.220749 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:31:06.220831 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:31:06.221343 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:31:06.221348 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:31:06.223258 (+ 1910us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.66.62:32879: Server oryx/127.9.66.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":79,"threads_started":1}
W20260925 07:31:06.223394 18684 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:31:06.218705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.3:43343 (local address 127.9.66.62:32879)
0925 07:31:06.218755 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 07:31:06.218756 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 07:31:06.218781 (+ 25us) server_negotiation.cc:407] Connection header received
0925 07:31:06.218873 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:31:06.218874 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:31:06.218885 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:31:06.218923 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:31:06.219202 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:31:06.219768 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:31:06.220439 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:31:06.220675 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:31:06.220750 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:31:06.223332 (+ 2582us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.66.3:43343: BlockingRecv error: recv got EOF from 127.9.66.3:43343 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Sep 25 07:31:06 dist-test-slave-57fn krb5kdc[17646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.66.1@KRBTEST.COM for oryx/127.9.66.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 07:31:06 dist-test-slave-57fn krb5kdc[17646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.66.1@KRBTEST.COM for oryx/127.9.66.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 07:31:06.986726 18687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:31:06.981908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:32879 (local address 127.9.66.1:38733)
0925 07:31:06.982240 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0925 07:31:06.982251 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 07:31:06.982313 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:31:06.982446 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:31:06.982446 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:31:06.982603 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:31:06.982649 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:31:06.982651 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:31:06.983369 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:31:06.983370 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:31:06.983870 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:31:06.983872 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:31:06.984269 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:31:06.984270 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:31:06.984341 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:31:06.984896 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:31:06.984900 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:31:06.986668 (+ 1768us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.66.62:32879: Server oryx/127.9.66.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":184,"threads_started":1}
W20260925 07:31:06.986874 18688 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:31:06.982098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.1:38733 (local address 127.9.66.62:32879)
0925 07:31:06.982199 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0925 07:31:06.982201 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:31:06.982295 (+ 94us) server_negotiation.cc:407] Connection header received
0925 07:31:06.982358 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:31:06.982359 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:31:06.982368 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:31:06.982420 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:31:06.982719 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:31:06.983313 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:31:06.983938 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:31:06.984167 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:31:06.984231 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:31:06.986778 (+ 2547us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.66.1:38733: BlockingRecv error: recv got EOF from 127.9.66.1:38733 (error 108)
Metrics: {"server-negotiator.queue_time_us":77,"thread_start_us":47,"threads_started":1}
Sep 25 07:31:07 dist-test-slave-57fn krb5kdc[17646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.66.2@KRBTEST.COM for oryx/127.9.66.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 07:31:07 dist-test-slave-57fn krb5kdc[17646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.66.2@KRBTEST.COM for oryx/127.9.66.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 07:31:07.104573 18689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:31:07.099830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:32879 (local address 127.9.66.2:40211)
0925 07:31:07.100024 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0925 07:31:07.100037 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 07:31:07.100113 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:31:07.100266 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:31:07.100267 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:31:07.100453 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:31:07.100500 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:31:07.100502 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:31:07.101187 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:31:07.101188 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:31:07.101810 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:31:07.101812 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:31:07.102192 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:31:07.102194 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:31:07.102249 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:31:07.102760 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:31:07.102765 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:31:07.104510 (+ 1745us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.66.62:32879: Server oryx/127.9.66.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":110,"threads_started":1}
W20260925 07:31:07.104673 18688 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:31:07.099931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.2:40211 (local address 127.9.66.62:32879)
0925 07:31:07.100011 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0925 07:31:07.100014 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:31:07.100109 (+ 95us) server_negotiation.cc:407] Connection header received
0925 07:31:07.100170 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:31:07.100171 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:31:07.100184 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:31:07.100260 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:31:07.100591 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:31:07.101100 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:31:07.101891 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:31:07.102116 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:31:07.102195 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:31:07.104605 (+ 2410us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.66.2:40211: BlockingRecv error: recv got EOF from 127.9.66.2:40211 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 25 07:31:07 dist-test-slave-57fn krb5kdc[17646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.66.3@KRBTEST.COM for oryx/127.9.66.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 07:31:07 dist-test-slave-57fn krb5kdc[17646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.66.3@KRBTEST.COM for oryx/127.9.66.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 07:31:07.228018 18690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:31:07.223803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:32879 (local address 127.9.66.3:43089)
0925 07:31:07.223925 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0925 07:31:07.223936 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 07:31:07.223997 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:31:07.224121 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:31:07.224122 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:31:07.224273 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:31:07.224319 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:31:07.224320 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:31:07.224911 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:31:07.224911 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:31:07.225416 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:31:07.225418 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:31:07.225729 (+ 311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:31:07.225730 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:31:07.225770 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:31:07.226193 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:31:07.226197 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:31:07.227960 (+ 1763us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.66.62:32879: Server oryx/127.9.66.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":72,"threads_started":1}
W20260925 07:31:07.228096 18688 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:31:07.223880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.3:43089 (local address 127.9.66.62:32879)
0925 07:31:07.223933 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 07:31:07.223935 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:31:07.223997 (+ 62us) server_negotiation.cc:407] Connection header received
0925 07:31:07.224037 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:31:07.224038 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:31:07.224049 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:31:07.224104 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:31:07.224375 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:31:07.224848 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:31:07.225479 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:31:07.225672 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:31:07.225721 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:31:07.228047 (+ 2326us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.66.3:43089: BlockingRecv error: recv got EOF from 127.9.66.3:43089 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Sep 25 07:31:07 dist-test-slave-57fn krb5kdc[17646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.66.1@KRBTEST.COM for oryx/127.9.66.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 07:31:07 dist-test-slave-57fn krb5kdc[17646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.66.1@KRBTEST.COM for oryx/127.9.66.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 07:31:07.991608 18691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:31:07.987111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:32879 (local address 127.9.66.1:39791)
0925 07:31:07.987315 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0925 07:31:07.987327 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 07:31:07.987394 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:31:07.987543 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:31:07.987543 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:31:07.987709 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:31:07.987761 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:31:07.987762 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:31:07.988429 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:31:07.988430 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:31:07.988945 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:31:07.988947 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:31:07.989265 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:31:07.989266 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:31:07.989303 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:31:07.989880 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:31:07.989884 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:31:07.991519 (+ 1635us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.66.62:32879: Server oryx/127.9.66.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":77,"threads_started":1}
W20260925 07:31:07.991691 18692 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:31:07.987268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.1:39791 (local address 127.9.66.62:32879)
0925 07:31:07.987395 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0925 07:31:07.987397 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:31:07.987405 (+ 8us) server_negotiation.cc:407] Connection header received
0925 07:31:07.987451 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:31:07.987452 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:31:07.987461 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:31:07.987517 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:31:07.987827 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:31:07.988332 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:31:07.989015 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:31:07.989207 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:31:07.989251 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:31:07.991600 (+ 2349us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.66.1:39791: BlockingRecv error: recv got EOF from 127.9.66.1:39791 (error 108)
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":74,"threads_started":1}
Sep 25 07:31:08 dist-test-slave-57fn krb5kdc[17646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.66.2@KRBTEST.COM for oryx/127.9.66.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 07:31:08 dist-test-slave-57fn krb5kdc[17646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.66.2@KRBTEST.COM for oryx/127.9.66.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 07:31:08.109829 18693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:31:08.105063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:32879 (local address 127.9.66.2:44447)
0925 07:31:08.105263 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0925 07:31:08.105277 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 07:31:08.105369 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:31:08.105627 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:31:08.105628 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:31:08.105814 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:31:08.105861 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:31:08.105863 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:31:08.106542 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:31:08.106543 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:31:08.107064 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:31:08.107066 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:31:08.107442 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:31:08.107444 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:31:08.107502 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:31:08.107933 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:31:08.107938 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:31:08.109718 (+ 1780us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.66.62:32879: Server oryx/127.9.66.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":107,"threads_started":1}
W20260925 07:31:08.109866 18692 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:31:08.105177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.2:44447 (local address 127.9.66.62:32879)
0925 07:31:08.105268 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0925 07:31:08.105270 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:31:08.105350 (+ 80us) server_negotiation.cc:407] Connection header received
0925 07:31:08.105462 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:31:08.105462 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:31:08.105475 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:31:08.105614 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:31:08.105943 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:31:08.106438 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:31:08.107163 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:31:08.107360 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:31:08.107434 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:31:08.109800 (+ 2366us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.66.2:44447: BlockingRecv error: recv got EOF from 127.9.66.2:44447 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 25 07:31:08 dist-test-slave-57fn krb5kdc[17646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.66.3@KRBTEST.COM for oryx/127.9.66.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 07:31:08 dist-test-slave-57fn krb5kdc[17646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.66.3@KRBTEST.COM for oryx/127.9.66.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 07:31:08.232813 18694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:31:08.228608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:32879 (local address 127.9.66.3:58489)
0925 07:31:08.228759 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 07:31:08.228770 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 07:31:08.228831 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:31:08.228950 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:31:08.228951 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:31:08.229115 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:31:08.229161 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:31:08.229163 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:31:08.229820 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:31:08.229820 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:31:08.230347 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:31:08.230349 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:31:08.230659 (+ 310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:31:08.230660 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:31:08.230698 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:31:08.231123 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:31:08.231128 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:31:08.232746 (+ 1618us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.66.62:32879: Server oryx/127.9.66.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":97,"threads_started":1}
W20260925 07:31:08.232874 18692 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:31:08.228647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.3:58489 (local address 127.9.66.62:32879)
0925 07:31:08.228693 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 07:31:08.228700 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 07:31:08.228831 (+ 131us) server_negotiation.cc:407] Connection header received
0925 07:31:08.228871 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:31:08.228872 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:31:08.228882 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:31:08.228932 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:31:08.229218 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:31:08.229758 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:31:08.230412 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:31:08.230599 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:31:08.230648 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:31:08.232828 (+ 2180us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.66.3:58489: BlockingRecv error: recv got EOF from 127.9.66.3:58489 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
Sep 25 07:31:08 dist-test-slave-57fn krb5kdc[17646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.66.1@KRBTEST.COM for oryx/127.9.66.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 07:31:09 dist-test-slave-57fn krb5kdc[17646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.66.1@KRBTEST.COM for oryx/127.9.66.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 07:31:09.002426 18695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:31:08.992086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:32879 (local address 127.9.66.1:56643)
0925 07:31:08.992238 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 07:31:08.992248 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 07:31:08.992300 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:31:08.992433 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:31:08.992434 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:31:08.992596 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:31:08.992643 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:31:08.992645 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:31:08.993262 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:31:08.993263 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:31:08.993783 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:31:08.993784 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:31:08.994088 (+ 304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:31:08.994088 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:31:08.994126 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:31:09.000569 (+ 6443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:31:09.000574 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:31:09.002350 (+ 1776us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.66.62:32879: Server oryx/127.9.66.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":84,"threads_started":1}
W20260925 07:31:09.002530 18696 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:31:08.992169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.1:56643 (local address 127.9.66.62:32879)
0925 07:31:08.992282 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0925 07:31:08.992284 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:31:08.992292 (+ 8us) server_negotiation.cc:407] Connection header received
0925 07:31:08.992353 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:31:08.992354 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:31:08.992363 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:31:08.992418 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:31:08.992704 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:31:08.993190 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:31:08.993846 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:31:08.994029 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:31:08.994079 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:31:09.002455 (+ 8376us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.66.1:56643: BlockingRecv error: recv got EOF from 127.9.66.1:56643 (error 108)
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":51,"threads_started":1}
Sep 25 07:31:09 dist-test-slave-57fn krb5kdc[17646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.66.2@KRBTEST.COM for oryx/127.9.66.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 07:31:09 dist-test-slave-57fn krb5kdc[17646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.66.2@KRBTEST.COM for oryx/127.9.66.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 07:31:09.114861 18697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:31:09.110299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:32879 (local address 127.9.66.2:57133)
0925 07:31:09.110419 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0925 07:31:09.110430 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 07:31:09.110506 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:31:09.110645 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:31:09.110646 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:31:09.110824 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:31:09.110870 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:31:09.110872 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:31:09.111524 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:31:09.111525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:31:09.112075 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:31:09.112077 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:31:09.112441 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:31:09.112442 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:31:09.112499 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:31:09.112955 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:31:09.112960 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:31:09.114793 (+ 1833us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.66.62:32879: Server oryx/127.9.66.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":71,"threads_started":1}
W20260925 07:31:09.114959 18696 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:31:09.110453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.2:57133 (local address 127.9.66.62:32879)
0925 07:31:09.110519 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 07:31:09.110521 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:31:09.110530 (+ 9us) server_negotiation.cc:407] Connection header received
0925 07:31:09.110558 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:31:09.110559 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:31:09.110570 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:31:09.110636 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:31:09.110947 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:31:09.111421 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:31:09.112160 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:31:09.112361 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:31:09.112438 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:31:09.114873 (+ 2435us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.66.2:57133: BlockingRecv error: recv got EOF from 127.9.66.2:57133 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 25 07:31:09 dist-test-slave-57fn krb5kdc[17646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.66.3@KRBTEST.COM for oryx/127.9.66.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 07:31:09 dist-test-slave-57fn krb5kdc[17646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.66.3@KRBTEST.COM for oryx/127.9.66.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 07:31:09.237582 18698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:31:09.233296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:32879 (local address 127.9.66.3:36407)
0925 07:31:09.233461 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 07:31:09.233472 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 07:31:09.233568 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:31:09.233744 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:31:09.233745 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:31:09.233912 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:31:09.233961 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:31:09.233962 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:31:09.234568 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:31:09.234568 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:31:09.235089 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:31:09.235090 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:31:09.235387 (+ 297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:31:09.235387 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:31:09.235425 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:31:09.235856 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:31:09.235861 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:31:09.237469 (+ 1608us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.66.62:32879: Server oryx/127.9.66.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":96,"threads_started":1}
W20260925 07:31:09.237651 18696 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:31:09.233367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.3:36407 (local address 127.9.66.62:32879)
0925 07:31:09.233428 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 07:31:09.233430 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:31:09.233548 (+ 118us) server_negotiation.cc:407] Connection header received
0925 07:31:09.233662 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:31:09.233663 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:31:09.233672 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:31:09.233720 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:31:09.234018 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:31:09.234500 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:31:09.235153 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:31:09.235326 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:31:09.235373 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:31:09.237605 (+ 2232us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.66.3:36407: BlockingRecv error: recv got EOF from 127.9.66.3:36407 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Sep 25 07:31:10 dist-test-slave-57fn krb5kdc[17646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.66.1@KRBTEST.COM for oryx/127.9.66.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 07:31:10 dist-test-slave-57fn krb5kdc[17646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.66.1@KRBTEST.COM for oryx/127.9.66.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 07:31:10.007508 18699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:31:10.002908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:32879 (local address 127.9.66.1:44953)
0925 07:31:10.003053 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 07:31:10.003063 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 07:31:10.003113 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:31:10.003322 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:31:10.003323 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:31:10.003480 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:31:10.003523 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:31:10.003525 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:31:10.004204 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:31:10.004204 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:31:10.004736 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:31:10.004737 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:31:10.005071 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:31:10.005072 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:31:10.005109 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:31:10.005609 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:31:10.005613 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:31:10.007411 (+ 1798us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.66.62:32879: Server oryx/127.9.66.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":84,"threads_started":1}
W20260925 07:31:10.007591 18700 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:31:10.003063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.1:44953 (local address 127.9.66.62:32879)
0925 07:31:10.003189 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0925 07:31:10.003191 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:31:10.003202 (+ 11us) server_negotiation.cc:407] Connection header received
0925 07:31:10.003228 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:31:10.003229 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:31:10.003238 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:31:10.003303 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:31:10.003596 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:31:10.004135 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:31:10.004808 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:31:10.005006 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:31:10.005059 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:31:10.007517 (+ 2458us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.66.1:44953: BlockingRecv error: recv got EOF from 127.9.66.1:44953 (error 108)
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":76,"threads_started":1}
Sep 25 07:31:10 dist-test-slave-57fn krb5kdc[17646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.66.2@KRBTEST.COM for oryx/127.9.66.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 07:31:10 dist-test-slave-57fn krb5kdc[17646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.66.2@KRBTEST.COM for oryx/127.9.66.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 07:31:10.120257 18701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:31:10.115405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:32879 (local address 127.9.66.2:38991)
0925 07:31:10.115613 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0925 07:31:10.115624 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 07:31:10.115705 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:31:10.115854 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:31:10.115855 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:31:10.116050 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:31:10.116095 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:31:10.116097 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:31:10.116837 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:31:10.116838 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:31:10.117362 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:31:10.117364 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:31:10.117809 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:31:10.117810 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:31:10.117868 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:31:10.118378 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:31:10.118385 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:31:10.120148 (+ 1763us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.66.62:32879: Server oryx/127.9.66.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":87,"threads_started":1}
W20260925 07:31:10.120285 18700 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:31:10.115483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.2:38991 (local address 127.9.66.62:32879)
0925 07:31:10.115534 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 07:31:10.115536 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:31:10.115701 (+ 165us) server_negotiation.cc:407] Connection header received
0925 07:31:10.115779 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:31:10.115780 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:31:10.115792 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:31:10.115844 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:31:10.116179 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:31:10.116750 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:31:10.117463 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:31:10.117725 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:31:10.117808 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:31:10.120223 (+ 2415us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.66.2:38991: BlockingRecv error: recv got EOF from 127.9.66.2:38991 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Sep 25 07:31:10 dist-test-slave-57fn krb5kdc[17646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.66.3@KRBTEST.COM for oryx/127.9.66.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 07:31:10 dist-test-slave-57fn krb5kdc[17646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.66.3@KRBTEST.COM for oryx/127.9.66.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 07:31:10.242547 18702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:31:10.238073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:32879 (local address 127.9.66.3:42687)
0925 07:31:10.238225 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 07:31:10.238236 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 07:31:10.238293 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:31:10.238413 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:31:10.238413 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:31:10.238584 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:31:10.238653 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:31:10.238654 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:31:10.239268 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:31:10.239269 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:31:10.239795 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:31:10.239796 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:31:10.240078 (+ 282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:31:10.240079 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:31:10.240115 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:31:10.240729 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:31:10.240733 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:31:10.242473 (+ 1740us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.66.62:32879: Server oryx/127.9.66.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":83,"threads_started":1}
W20260925 07:31:10.242643 18700 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:31:10.238128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.3:42687 (local address 127.9.66.62:32879)
0925 07:31:10.238173 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 07:31:10.238175 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:31:10.238276 (+ 101us) server_negotiation.cc:407] Connection header received
0925 07:31:10.238332 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:31:10.238333 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:31:10.238343 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:31:10.238388 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:31:10.238722 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:31:10.239203 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:31:10.239859 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:31:10.240021 (+ 162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:31:10.240066 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:31:10.242553 (+ 2487us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.66.3:42687: BlockingRecv error: recv got EOF from 127.9.66.3:42687 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
Sep 25 07:31:11 dist-test-slave-57fn krb5kdc[17646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.66.1@KRBTEST.COM for oryx/127.9.66.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 07:31:11 dist-test-slave-57fn krb5kdc[17646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.66.1@KRBTEST.COM for oryx/127.9.66.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 07:31:11.012238 18703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:31:11.008004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:32879 (local address 127.9.66.1:53793)
0925 07:31:11.008142 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 07:31:11.008153 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 07:31:11.008210 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:31:11.008336 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:31:11.008336 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:31:11.008492 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:31:11.008538 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:31:11.008539 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:31:11.009156 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:31:11.009156 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:31:11.009710 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:31:11.009711 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:31:11.010019 (+ 308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:31:11.010020 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:31:11.010056 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:31:11.010481 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:31:11.010485 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:31:11.012180 (+ 1695us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.66.62:32879: Server oryx/127.9.66.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":84,"threads_started":1}
W20260925 07:31:11.012347 18704 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:31:11.008089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.1:53793 (local address 127.9.66.62:32879)
0925 07:31:11.008220 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0925 07:31:11.008222 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:31:11.008229 (+ 7us) server_negotiation.cc:407] Connection header received
0925 07:31:11.008252 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:31:11.008253 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:31:11.008262 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:31:11.008321 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:31:11.008602 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:31:11.009091 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:31:11.009777 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:31:11.009961 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:31:11.010011 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:31:11.012266 (+ 2255us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.66.1:53793: BlockingRecv error: recv got EOF from 127.9.66.1:53793 (error 108)
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":76,"threads_started":1}
Sep 25 07:31:11 dist-test-slave-57fn krb5kdc[17646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.66.2@KRBTEST.COM for oryx/127.9.66.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 07:31:11 dist-test-slave-57fn krb5kdc[17646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.66.2@KRBTEST.COM for oryx/127.9.66.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 07:31:11.125089 18705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:31:11.120770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:32879 (local address 127.9.66.2:53547)
0925 07:31:11.120925 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 07:31:11.120936 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 07:31:11.121001 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:31:11.121139 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:31:11.121140 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:31:11.121297 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:31:11.121344 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:31:11.121346 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:31:11.121958 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:31:11.121959 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:31:11.122476 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:31:11.122478 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:31:11.122777 (+ 299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:31:11.122777 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:31:11.122815 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:31:11.123364 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:31:11.123369 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:31:11.125026 (+ 1657us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.66.62:32879: Server oryx/127.9.66.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":92,"threads_started":1}
W20260925 07:31:11.125207 18704 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:31:11.120835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.2:53547 (local address 127.9.66.62:32879)
0925 07:31:11.120910 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0925 07:31:11.120912 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:31:11.120979 (+ 67us) server_negotiation.cc:407] Connection header received
0925 07:31:11.121055 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:31:11.121056 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:31:11.121065 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:31:11.121118 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:31:11.121406 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:31:11.121899 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:31:11.122544 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:31:11.122717 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:31:11.122765 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:31:11.125114 (+ 2349us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.66.2:53547: BlockingRecv error: recv got EOF from 127.9.66.2:53547 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 07:31:11.165948 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 18229
I20260925 07:31:11.170508 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 18381
I20260925 07:31:11.174765 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 18513
I20260925 07:31:11.179064 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 18159
2026-09-25T07:31:11Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17392 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-25T07:31:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T07:31:11Z Disabled control of system clock
I20260925 07:31:11.206907 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790321392278483-9480-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790321392278483-9480-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790321392278483-9480-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.66.62:36111
--webserver_interface=127.9.66.62
--webserver_port=0
--builtin_ntp_servers=127.9.66.20:34921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.66.62:36111
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260925 07:31:11.290795 18709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:31:11.290999 18709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:31:11.291020 18709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:31:11.292210 18709 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:31:11.292258 18709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:31:11.292273 18709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:31:11.292287 18709 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:31:11.292301 18709 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:31:11.293684 18709 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:34921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790321392278483-9480-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790321392278483-9480-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.66.62:36111
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.66.62:36111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.66.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18709
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790321392278483-9480-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:31:11.293938 18709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:31:11.294256 18709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:31:11.298952 18717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:31:11.299048 18709 server_base.cc:1034] running on GCE node
W20260925 07:31:11.300635 18715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:31:11.300762 18714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:31:11.301024 18709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:31:11.301251 18709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:31:11.302417 18709 hybrid_clock.cc:648] HybridClock initialized: now 1790321471302405 us; error 24 us; skew 500 ppm
I20260925 07:31:11.303687 18709 webserver.cc:460] Webserver started at http://127.9.66.62:37963/ using document root <none> and password file <none>
I20260925 07:31:11.303922 18709 fs_manager.cc:359] Metadata directory not provided
I20260925 07:31:11.303992 18709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:31:11.304142 18709 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:31:11.305117 18709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790321392278483-9480-0/minicluster-data/master-0/data/instance:
uuid: "e16ca00ed2fc48bd8522f4899341b72e"
format_stamp: "Formatted at 2026-09-25 07:31:11 on dist-test-slave-57fn"
server_key: "27eef27918b4fc1edb12ffc5d2f7cc4e"
server_key_iv: "3e90afb2c5df733f97ad08e10ba2c35e"
server_key_version: "encryptionkey@0"
I20260925 07:31:11.305470 18709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790321392278483-9480-0/minicluster-data/master-0/wal/instance:
uuid: "e16ca00ed2fc48bd8522f4899341b72e"
format_stamp: "Formatted at 2026-09-25 07:31:11 on dist-test-slave-57fn"
server_key: "27eef27918b4fc1edb12ffc5d2f7cc4e"
server_key_iv: "3e90afb2c5df733f97ad08e10ba2c35e"
server_key_version: "encryptionkey@0"
I20260925 07:31:11.306772 18709 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 07:31:11.307457 18723 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:31:11.307628 18709 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 07:31:11.307721 18709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790321392278483-9480-0/minicluster-data/master-0/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790321392278483-9480-0/minicluster-data/master-0/wal
uuid: "e16ca00ed2fc48bd8522f4899341b72e"
format_stamp: "Formatted at 2026-09-25 07:31:11 on dist-test-slave-57fn"
server_key: "27eef27918b4fc1edb12ffc5d2f7cc4e"
server_key_iv: "3e90afb2c5df733f97ad08e10ba2c35e"
server_key_version: "encryptionkey@0"
I20260925 07:31:11.307801 18709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790321392278483-9480-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790321392278483-9480-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790321392278483-9480-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:31:11.329909 18709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:31:11.330240 18709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:31:11.330397 18709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:31:11.334363 18709 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.62:36111
I20260925 07:31:11.334408 18775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.62:36111 every 8 connection(s)
I20260925 07:31:11.334770 18709 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
I20260925 07:31:11.335414 18776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:31:11.337875 18776 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e16ca00ed2fc48bd8522f4899341b72e: Bootstrap starting.
I20260925 07:31:11.338378 18776 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e16ca00ed2fc48bd8522f4899341b72e: Neither blocks nor log segments found. Creating new log.
I20260925 07:31:11.338608 18776 log.cc:824] T 00000000000000000000000000000000 P e16ca00ed2fc48bd8522f4899341b72e: Log is configured to *not* fsync() on all Append() calls
I20260925 07:31:11.339198 18776 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e16ca00ed2fc48bd8522f4899341b72e: No bootstrap required, opened a new log
I20260925 07:31:11.340657 18776 raft_consensus.cc:348] T 00000000000000000000000000000000 P e16ca00ed2fc48bd8522f4899341b72e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e16ca00ed2fc48bd8522f4899341b72e" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 36111 } }
I20260925 07:31:11.340787 18776 raft_consensus.cc:374] T 00000000000000000000000000000000 P e16ca00ed2fc48bd8522f4899341b72e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:31:11.340808 18776 raft_consensus.cc:729] T 00000000000000000000000000000000 P e16ca00ed2fc48bd8522f4899341b72e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e16ca00ed2fc48bd8522f4899341b72e, State: Initialized, Role: FOLLOWER
I20260925 07:31:11.340937 18776 consensus_queue.cc:260] T 00000000000000000000000000000000 P e16ca00ed2fc48bd8522f4899341b72e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e16ca00ed2fc48bd8522f4899341b72e" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 36111 } }
I20260925 07:31:11.341018 18776 raft_consensus.cc:388] T 00000000000000000000000000000000 P e16ca00ed2fc48bd8522f4899341b72e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 07:31:11.341041 18776 raft_consensus.cc:482] T 00000000000000000000000000000000 P e16ca00ed2fc48bd8522f4899341b72e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 07:31:11.341114 18776 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e16ca00ed2fc48bd8522f4899341b72e [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:31:11.341084 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 18709
I20260925 07:31:11.341185 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790321392278483-9480-0/minicluster-data/master-0/wal/instance
I20260925 07:31:11.341380 9480 external_mini_cluster.cc:1428] Setting key 0dc4d853329ed634f138d5eff8dde664
I20260925 07:31:11.341810 18776 raft_consensus.cc:504] T 00000000000000000000000000000000 P e16ca00ed2fc48bd8522f4899341b72e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e16ca00ed2fc48bd8522f4899341b72e" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 36111 } }
I20260925 07:31:11.341928 18776 leader_election.cc:302] T 00000000000000000000000000000000 P e16ca00ed2fc48bd8522f4899341b72e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e16ca00ed2fc48bd8522f4899341b72e; no voters:
I20260925 07:31:11.342110 18776 leader_election.cc:288] T 00000000000000000000000000000000 P e16ca00ed2fc48bd8522f4899341b72e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 07:31:11.342149 18781 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e16ca00ed2fc48bd8522f4899341b72e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:31:11.342308 18781 raft_consensus.cc:686] T 00000000000000000000000000000000 P e16ca00ed2fc48bd8522f4899341b72e [term 1 LEADER]: Becoming Leader. State: Replica: e16ca00ed2fc48bd8522f4899341b72e, State: Running, Role: LEADER
I20260925 07:31:11.342346 18776 sys_catalog.cc:564] T 00000000000000000000000000000000 P e16ca00ed2fc48bd8522f4899341b72e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:31:11.342413 18781 consensus_queue.cc:237] T 00000000000000000000000000000000 P e16ca00ed2fc48bd8522f4899341b72e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e16ca00ed2fc48bd8522f4899341b72e" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 36111 } }
I20260925 07:31:11.342870 18783 sys_catalog.cc:455] T 00000000000000000000000000000000 P e16ca00ed2fc48bd8522f4899341b72e [sys.catalog]: SysCatalogTable state changed. Reason: New leader e16ca00ed2fc48bd8522f4899341b72e. Latest consensus state: current_term: 1 leader_uuid: "e16ca00ed2fc48bd8522f4899341b72e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e16ca00ed2fc48bd8522f4899341b72e" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 36111 } } }
I20260925 07:31:11.342947 18783 sys_catalog.cc:458] T 00000000000000000000000000000000 P e16ca00ed2fc48bd8522f4899341b72e [sys.catalog]: This master's current role is: LEADER
I20260925 07:31:11.343005 18782 sys_catalog.cc:455] T 00000000000000000000000000000000 P e16ca00ed2fc48bd8522f4899341b72e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e16ca00ed2fc48bd8522f4899341b72e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e16ca00ed2fc48bd8522f4899341b72e" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 36111 } } }
I20260925 07:31:11.343086 18782 sys_catalog.cc:458] T 00000000000000000000000000000000 P e16ca00ed2fc48bd8522f4899341b72e [sys.catalog]: This master's current role is: LEADER
W20260925 07:31:11.344208 18797 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e16ca00ed2fc48bd8522f4899341b72e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 07:31:11.344261 18797 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 07:31:11.344303 18790 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 07:31:11.344430 18790 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 07:31:11.345906 18790 catalog_manager.cc:1347] Generated new cluster ID: 703fcd989ecf4af88bdbf50e468c430c
I20260925 07:31:11.345969 18790 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 07:31:11.347291 18780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:31:11.341912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50122 (local address 127.9.66.62:36111)
0925 07:31:11.342122 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0925 07:31:11.342132 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 07:31:11.342164 (+ 32us) server_negotiation.cc:407] Connection header received
0925 07:31:11.342290 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:31:11.342298 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:31:11.342371 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:31:11.342772 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:31:11.342869 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:31:11.344280 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:31:11.345186 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:31:11.345791 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:31:11.345983 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:31:11.346688 (+ 705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:31:11.346692 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:31:11.346692 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 07:31:11.346860 (+ 168us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:31:11.346889 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 07:31:11.346899 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:31:11.346969 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 07:31:11.347051 (+ 82us) server_negotiation.cc:299] Negotiation successful
0925 07:31:11.347140 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":79,"threads_started":1}
I20260925 07:31:11.353111 18790 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 07:31:11.353785 18790 catalog_manager.cc:1504] Loading token signing keys...
I20260925 07:31:11.366515 18790 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e16ca00ed2fc48bd8522f4899341b72e: Generated new TSK 0
I20260925 07:31:11.366699 18790 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 07:31:11.372023 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790321392278483-9480-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790321392278483-9480-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790321392278483-9480-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790321392278483-9480-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.66.1:0
--local_ip_for_outbound_sockets=127.9.66.1
--webserver_interface=127.9.66.1
--webserver_port=0
--tserver_master_addrs=127.9.66.62:36111
--builtin_ntp_servers=127.9.66.20:34921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260925 07:31:11.463246 18800 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:31:11.463439 18800 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:31:11.463460 18800 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:31:11.464869 18800 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:31:11.464928 18800 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:31:11.464957 18800 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.66.1
I20260925 07:31:11.466420 18800 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:34921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790321392278483-9480-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790321392278483-9480-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.66.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790321392278483-9480-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.66.1
--webserver_port=0
--tserver_master_addrs=127.9.66.62:36111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18800
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.66.1
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790321392278483-9480-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:31:11.466671 18800 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:31:11.466985 18800 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:31:11.472185 18805 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:31:11.472307 18808 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:31:11.472200 18806 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:31:11.472612 18800 server_base.cc:1034] running on GCE node
I20260925 07:31:11.472822 18800 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:31:11.473093 18800 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:31:11.474260 18800 hybrid_clock.cc:648] HybridClock initialized: now 1790321471474251 us; error 33 us; skew 500 ppm
I20260925 07:31:11.475658 18800 webserver.cc:460] Webserver started at http://127.9.66.1:40773/ using document root <none> and password file <none>
I20260925 07:31:11.475915 18800 fs_manager.cc:359] Metadata directory not provided
I20260925 07:31:11.475970 18800 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:31:11.476092 18800 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:31:11.477447 18800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790321392278483-9480-0/minicluster-data/ts-0/data/instance:
uuid: "4b350282908c4e74aa26ae7519616060"
format_stamp: "Formatted at 2026-09-25 07:31:11 on dist-test-slave-57fn"
server_key: "473622f95af8d2fc47534559241e227c"
server_key_iv: "f3284418ad58df5eb9ef9dbf141acdd2"
server_key_version: "encryptionkey@0"
I20260925 07:31:11.477872 18800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790321392278483-9480-0/minicluster-data/ts-0/wal/instance:
uuid: "4b350282908c4e74aa26ae7519616060"
format_stamp: "Formatted at 2026-09-25 07:31:11 on dist-test-slave-57fn"
server_key: "473622f95af8d2fc47534559241e227c"
server_key_iv: "f3284418ad58df5eb9ef9dbf141acdd2"
server_key_version: "encryptionkey@0"
I20260925 07:31:11.479422 18800 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 07:31:11.480283 18814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:31:11.480521 18800 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:31:11.480598 18800 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790321392278483-9480-0/minicluster-data/ts-0/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790321392278483-9480-0/minicluster-data/ts-0/wal
uuid: "4b350282908c4e74aa26ae7519616060"
format_stamp: "Formatted at 2026-09-25 07:31:11 on dist-test-slave-57fn"
server_key: "473622f95af8d2fc47534559241e227c"
server_key_iv: "f3284418ad58df5eb9ef9dbf141acdd2"
server_key_version: "encryptionkey@0"
I20260925 07:31:11.480684 18800 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790321392278483-9480-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790321392278483-9480-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790321392278483-9480-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:31:11.503026 18800 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:31:11.503342 18800 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:31:11.503485 18800 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:31:11.503701 18800 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:31:11.504071 18800 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:31:11.504118 18800 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:31:11.504160 18800 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:31:11.504189 18800 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:31:11.510643 18800 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.1:45401
I20260925 07:31:11.510682 18927 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.1:45401 every 8 connection(s)
I20260925 07:31:11.511008 18800 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790321392278483-9480-0/minicluster-data/ts-0/data/info.pb
I20260925 07:31:11.515774 18780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:31:11.511729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.1:45563 (local address 127.9.66.62:36111)
0925 07:31:11.511770 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 07:31:11.511772 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:31:11.512014 (+ 242us) server_negotiation.cc:407] Connection header received
0925 07:31:11.512469 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:31:11.512470 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:31:11.512481 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:31:11.512540 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:31:11.513145 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:31:11.513671 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:31:11.514544 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:31:11.514745 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:31:11.514801 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:31:11.515555 (+ 754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:31:11.515555 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:31:11.515556 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 07:31:11.515559 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:31:11.515570 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 07:31:11.515573 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:31:11.515612 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0925 07:31:11.515694 (+ 82us) server_negotiation.cc:299] Negotiation successful
0925 07:31:11.515718 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 07:31:11.515947 18930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:31:11.511705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:36111 (local address 127.9.66.1:45563)
0925 07:31:11.511888 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0925 07:31:11.511926 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 07:31:11.512387 (+ 461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:31:11.512617 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:31:11.512621 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:31:11.512628 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:31:11.513075 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:31:11.513080 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:31:11.513737 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:31:11.513738 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:31:11.514457 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:31:11.514459 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:31:11.514814 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:31:11.514815 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:31:11.514912 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:31:11.515435 (+ 523us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 07:31:11.515436 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:31:11.515462 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 07:31:11.515483 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:31:11.515632 (+ 149us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:31:11.515635 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:31:11.515637 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 07:31:11.515686 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 07:31:11.515757 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":97,"threads_started":1}
I20260925 07:31:11.516408 18928 heartbeater.cc:347] Connected to a master server at 127.9.66.62:36111
I20260925 07:31:11.516491 18928 heartbeater.cc:464] Registering TS with master...
I20260925 07:31:11.516628 18928 heartbeater.cc:511] Master 127.9.66.62:36111 requested a full tablet report, sending...
I20260925 07:31:11.517086 18740 ts_manager.cc:194] Registered new tserver with Master: 4b350282908c4e74aa26ae7519616060 (127.9.66.1:45401)
I20260925 07:31:11.517633 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 18800
I20260925 07:31:11.517714 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790321392278483-9480-0/minicluster-data/ts-0/wal/instance
I20260925 07:31:11.517895 9480 external_mini_cluster.cc:1428] Setting key 6d1c08d370d2f8d66d796f730e340856
I20260925 07:31:11.517978 18740 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.66.1:45563
I20260925 07:31:11.518808 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790321392278483-9480-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790321392278483-9480-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790321392278483-9480-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790321392278483-9480-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.66.2:0
--local_ip_for_outbound_sockets=127.9.66.2
--webserver_interface=127.9.66.2
--webserver_port=0
--tserver_master_addrs=127.9.66.62:36111
--builtin_ntp_servers=127.9.66.20:34921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260925 07:31:11.597137 18931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:31:11.597326 18931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:31:11.597347 18931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:31:11.598441 18931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:31:11.598475 18931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:31:11.598500 18931 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.66.2
I20260925 07:31:11.599704 18931 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:34921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790321392278483-9480-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790321392278483-9480-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.66.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790321392278483-9480-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.66.2
--webserver_port=0
--tserver_master_addrs=127.9.66.62:36111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18931
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.66.2
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790321392278483-9480-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:31:11.599939 18931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:31:11.600246 18931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:31:11.605041 18937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:31:11.605024 18939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:31:11.605072 18931 server_base.cc:1034] running on GCE node
W20260925 07:31:11.605024 18936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:31:11.605429 18931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:31:11.605720 18931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:31:11.606873 18931 hybrid_clock.cc:648] HybridClock initialized: now 1790321471606862 us; error 37 us; skew 500 ppm
I20260925 07:31:11.608121 18931 webserver.cc:460] Webserver started at http://127.9.66.2:46543/ using document root <none> and password file <none>
I20260925 07:31:11.608346 18931 fs_manager.cc:359] Metadata directory not provided
I20260925 07:31:11.608413 18931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:31:11.608556 18931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:31:11.609592 18931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790321392278483-9480-0/minicluster-data/ts-1/data/instance:
uuid: "c31e7ccd17154860afe4d3623bf29197"
format_stamp: "Formatted at 2026-09-25 07:31:11 on dist-test-slave-57fn"
server_key: "3f3790e8110072aaf6d75c606b0bbf99"
server_key_iv: "f3feb183151b1e2987402c58821b22a8"
server_key_version: "encryptionkey@0"
I20260925 07:31:11.609941 18931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790321392278483-9480-0/minicluster-data/ts-1/wal/instance:
uuid: "c31e7ccd17154860afe4d3623bf29197"
format_stamp: "Formatted at 2026-09-25 07:31:11 on dist-test-slave-57fn"
server_key: "3f3790e8110072aaf6d75c606b0bbf99"
server_key_iv: "f3feb183151b1e2987402c58821b22a8"
server_key_version: "encryptionkey@0"
I20260925 07:31:11.611227 18931 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 07:31:11.611929 18945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:31:11.612099 18931 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 07:31:11.612192 18931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790321392278483-9480-0/minicluster-data/ts-1/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790321392278483-9480-0/minicluster-data/ts-1/wal
uuid: "c31e7ccd17154860afe4d3623bf29197"
format_stamp: "Formatted at 2026-09-25 07:31:11 on dist-test-slave-57fn"
server_key: "3f3790e8110072aaf6d75c606b0bbf99"
server_key_iv: "f3feb183151b1e2987402c58821b22a8"
server_key_version: "encryptionkey@0"
I20260925 07:31:11.612274 18931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790321392278483-9480-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790321392278483-9480-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790321392278483-9480-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:31:11.623152 18931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:31:11.623396 18931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:31:11.623531 18931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:31:11.623728 18931 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:31:11.624010 18931 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:31:11.624061 18931 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:31:11.624091 18931 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:31:11.624116 18931 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:31:11.629456 18931 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.2:42535
I20260925 07:31:11.629576 19058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.2:42535 every 8 connection(s)
I20260925 07:31:11.629846 18931 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790321392278483-9480-0/minicluster-data/ts-1/data/info.pb
I20260925 07:31:11.632969 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 18931
I20260925 07:31:11.633066 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790321392278483-9480-0/minicluster-data/ts-1/wal/instance
I20260925 07:31:11.633245 9480 external_mini_cluster.cc:1428] Setting key 151dbac23b2a5880dcfd764a412195b3
I20260925 07:31:11.634212 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790321392278483-9480-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790321392278483-9480-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790321392278483-9480-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790321392278483-9480-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.66.3:0
--local_ip_for_outbound_sockets=127.9.66.3
--webserver_interface=127.9.66.3
--webserver_port=0
--tserver_master_addrs=127.9.66.62:36111
--builtin_ntp_servers=127.9.66.20:34921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260925 07:31:11.635116 18780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:31:11.630496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.2:50733 (local address 127.9.66.62:36111)
0925 07:31:11.630553 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 07:31:11.630556 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:31:11.630729 (+ 173us) server_negotiation.cc:407] Connection header received
0925 07:31:11.631237 (+ 508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:31:11.631238 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:31:11.631251 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:31:11.631301 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:31:11.631918 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:31:11.632396 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:31:11.633663 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:31:11.633889 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:31:11.633954 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:31:11.634882 (+ 928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:31:11.634883 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:31:11.634883 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 07:31:11.634887 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:31:11.634899 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 07:31:11.634903 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:31:11.634952 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 07:31:11.635042 (+ 90us) server_negotiation.cc:299] Negotiation successful
0925 07:31:11.635062 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 07:31:11.635382 19061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:31:11.630460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:36111 (local address 127.9.66.2:50733)
0925 07:31:11.630611 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 07:31:11.630650 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 07:31:11.631122 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:31:11.631362 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:31:11.631365 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:31:11.631375 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:31:11.631827 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:31:11.631831 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:31:11.632488 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:31:11.632489 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:31:11.633570 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:31:11.633573 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:31:11.633975 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:31:11.633976 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:31:11.634133 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:31:11.634740 (+ 607us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 07:31:11.634740 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:31:11.634775 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 07:31:11.634801 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:31:11.634971 (+ 170us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:31:11.634974 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:31:11.634976 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 07:31:11.635040 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 07:31:11.635145 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":62,"threads_started":1}
I20260925 07:31:11.635795 19059 heartbeater.cc:347] Connected to a master server at 127.9.66.62:36111
I20260925 07:31:11.635890 19059 heartbeater.cc:464] Registering TS with master...
I20260925 07:31:11.636065 19059 heartbeater.cc:511] Master 127.9.66.62:36111 requested a full tablet report, sending...
I20260925 07:31:11.636372 18740 ts_manager.cc:194] Registered new tserver with Master: c31e7ccd17154860afe4d3623bf29197 (127.9.66.2:42535)
I20260925 07:31:11.636729 18740 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.66.2:50733
W20260925 07:31:11.714149 19062 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:31:11.714336 19062 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:31:11.714366 19062 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:31:11.715570 19062 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:31:11.715612 19062 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:31:11.715668 19062 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.66.3
I20260925 07:31:11.716969 19062 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:34921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790321392278483-9480-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790321392278483-9480-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.66.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790321392278483-9480-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.66.3
--webserver_port=0
--tserver_master_addrs=127.9.66.62:36111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19062
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.66.3
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790321392278483-9480-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:31:11.717238 19062 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:31:11.717579 19062 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:31:11.722174 19067 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:31:11.722308 19070 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:31:11.722182 19068 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:31:11.722613 19062 server_base.cc:1034] running on GCE node
I20260925 07:31:11.722815 19062 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:31:11.723037 19062 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:31:11.724184 19062 hybrid_clock.cc:648] HybridClock initialized: now 1790321471724157 us; error 38 us; skew 500 ppm
I20260925 07:31:11.725641 19062 webserver.cc:460] Webserver started at http://127.9.66.3:40803/ using document root <none> and password file <none>
I20260925 07:31:11.725831 19062 fs_manager.cc:359] Metadata directory not provided
I20260925 07:31:11.725868 19062 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:31:11.725976 19062 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:31:11.726902 19062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790321392278483-9480-0/minicluster-data/ts-2/data/instance:
uuid: "dc823047d26b4de38e4e5b147daf23bd"
format_stamp: "Formatted at 2026-09-25 07:31:11 on dist-test-slave-57fn"
server_key: "303c116d0be83b4201814f34001d85e4"
server_key_iv: "d79d8e77cae2f267ec7d5e54cf0fe130"
server_key_version: "encryptionkey@0"
I20260925 07:31:11.727213 19062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790321392278483-9480-0/minicluster-data/ts-2/wal/instance:
uuid: "dc823047d26b4de38e4e5b147daf23bd"
format_stamp: "Formatted at 2026-09-25 07:31:11 on dist-test-slave-57fn"
server_key: "303c116d0be83b4201814f34001d85e4"
server_key_iv: "d79d8e77cae2f267ec7d5e54cf0fe130"
server_key_version: "encryptionkey@0"
I20260925 07:31:11.728431 19062 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 07:31:11.729121 19076 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:31:11.729315 19062 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 07:31:11.729436 19062 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790321392278483-9480-0/minicluster-data/ts-2/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790321392278483-9480-0/minicluster-data/ts-2/wal
uuid: "dc823047d26b4de38e4e5b147daf23bd"
format_stamp: "Formatted at 2026-09-25 07:31:11 on dist-test-slave-57fn"
server_key: "303c116d0be83b4201814f34001d85e4"
server_key_iv: "d79d8e77cae2f267ec7d5e54cf0fe130"
server_key_version: "encryptionkey@0"
I20260925 07:31:11.729538 19062 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790321392278483-9480-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790321392278483-9480-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790321392278483-9480-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:31:11.741293 19062 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:31:11.741619 19062 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:31:11.741787 19062 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:31:11.742002 19062 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:31:11.742298 19062 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:31:11.742353 19062 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:31:11.742460 19062 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:31:11.742512 19062 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:31:11.748813 19062 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.3:41739
I20260925 07:31:11.748848 19189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.3:41739 every 8 connection(s)
I20260925 07:31:11.749166 19062 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790321392278483-9480-0/minicluster-data/ts-2/data/info.pb
I20260925 07:31:11.753834 18780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:31:11.749901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.3:60919 (local address 127.9.66.62:36111)
0925 07:31:11.749946 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 07:31:11.749948 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:31:11.750175 (+ 227us) server_negotiation.cc:407] Connection header received
0925 07:31:11.750623 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:31:11.750624 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:31:11.750634 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:31:11.750682 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:31:11.751303 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:31:11.751801 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:31:11.752670 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:31:11.752829 (+ 159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:31:11.752872 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:31:11.753630 (+ 758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:31:11.753632 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:31:11.753632 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 07:31:11.753636 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:31:11.753648 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 07:31:11.753651 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:31:11.753690 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0925 07:31:11.753757 (+ 67us) server_negotiation.cc:299] Negotiation successful
0925 07:31:11.753778 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 07:31:11.754055 19192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:31:11.749892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:36111 (local address 127.9.66.3:60919)
0925 07:31:11.750061 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 07:31:11.750092 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0925 07:31:11.750527 (+ 435us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:31:11.750736 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:31:11.750739 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:31:11.750747 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:31:11.751201 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:31:11.751205 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:31:11.751881 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:31:11.751881 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:31:11.752600 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:31:11.752602 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:31:11.752911 (+ 309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:31:11.752912 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:31:11.753009 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:31:11.753454 (+ 445us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 07:31:11.753455 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:31:11.753479 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 07:31:11.753498 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:31:11.753693 (+ 195us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:31:11.753695 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:31:11.753698 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 07:31:11.753743 (+ 45us) client_negotiation.cc:240] Negotiation successful
0925 07:31:11.753813 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":81,"threads_started":1}
I20260925 07:31:11.754714 19190 heartbeater.cc:347] Connected to a master server at 127.9.66.62:36111
I20260925 07:31:11.754807 19190 heartbeater.cc:464] Registering TS with master...
I20260925 07:31:11.754967 19190 heartbeater.cc:511] Master 127.9.66.62:36111 requested a full tablet report, sending...
I20260925 07:31:11.755323 18740 ts_manager.cc:194] Registered new tserver with Master: dc823047d26b4de38e4e5b147daf23bd (127.9.66.3:41739)
I20260925 07:31:11.755667 18740 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.66.3:60919
I20260925 07:31:11.759222 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 19062
I20260925 07:31:11.759303 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790321392278483-9480-0/minicluster-data/ts-2/wal/instance
I20260925 07:31:11.759439 9480 external_mini_cluster.cc:1428] Setting key 1a163b4721c211682bab651e2a37afce
I20260925 07:31:11.760224 9480 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 07:31:11.761919 18780 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:31:11.761463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50124 (local address 127.9.66.62:36111)
0925 07:31:11.761538 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0925 07:31:11.761541 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:31:11.761653 (+ 112us) server_negotiation.cc:407] Connection header received
0925 07:31:11.761699 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:31:11.761699 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:31:11.761708 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:31:11.761753 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:31:11.761858 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50124: BlockingRecv error: recv got EOF from 127.0.0.1:50124 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 07:31:11.762043 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 18800
I20260925 07:31:11.766377 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 18931
I20260925 07:31:11.770558 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 19062
I20260925 07:31:11.774696 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 18709
2026-09-25T07:31:11Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (593 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-25T07:31:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T07:31:11Z Disabled control of system clock
I20260925 07:31:11.800261 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790321392278483-9480-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790321392278483-9480-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790321392278483-9480-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.66.62:40125
--webserver_interface=127.9.66.62
--webserver_port=0
--builtin_ntp_servers=127.9.66.20:35265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.66.62:40125
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 07:31:11.878579 19201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:31:11.878765 19201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:31:11.878786 19201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:31:11.879860 19201 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:31:11.879892 19201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:31:11.879905 19201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:31:11.879917 19201 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:31:11.879930 19201 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:31:11.881146 19201 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:35265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790321392278483-9480-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790321392278483-9480-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.66.62:40125
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.66.62:40125
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.66.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19201
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790321392278483-9480-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:31:11.881333 19201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:31:11.881554 19201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:31:11.886196 19206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:31:11.886209 19207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:31:11.886211 19209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:31:11.886353 19201 server_base.cc:1034] running on GCE node
I20260925 07:31:11.886612 19201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:31:11.886883 19201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:31:11.888020 19201 hybrid_clock.cc:648] HybridClock initialized: now 1790321471888014 us; error 28 us; skew 500 ppm
I20260925 07:31:11.889199 19201 webserver.cc:460] Webserver started at http://127.9.66.62:34401/ using document root <none> and password file <none>
I20260925 07:31:11.889424 19201 fs_manager.cc:359] Metadata directory not provided
I20260925 07:31:11.889467 19201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:31:11.889633 19201 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:31:11.890669 19201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790321392278483-9480-0/minicluster-data/master-0/data/instance:
uuid: "79ce6a9d28a8493aaa389dc4c10435c2"
format_stamp: "Formatted at 2026-09-25 07:31:11 on dist-test-slave-57fn"
server_key: "e769d0304998d8231ad757b1325d7a26"
server_key_iv: "9d62681509ebe62e641a3e8530468a6e"
server_key_version: "encryptionkey@0"
I20260925 07:31:11.891005 19201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790321392278483-9480-0/minicluster-data/master-0/wal/instance:
uuid: "79ce6a9d28a8493aaa389dc4c10435c2"
format_stamp: "Formatted at 2026-09-25 07:31:11 on dist-test-slave-57fn"
server_key: "e769d0304998d8231ad757b1325d7a26"
server_key_iv: "9d62681509ebe62e641a3e8530468a6e"
server_key_version: "encryptionkey@0"
I20260925 07:31:11.892249 19201 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 07:31:11.893002 19215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:31:11.893203 19201 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 07:31:11.893287 19201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790321392278483-9480-0/minicluster-data/master-0/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790321392278483-9480-0/minicluster-data/master-0/wal
uuid: "79ce6a9d28a8493aaa389dc4c10435c2"
format_stamp: "Formatted at 2026-09-25 07:31:11 on dist-test-slave-57fn"
server_key: "e769d0304998d8231ad757b1325d7a26"
server_key_iv: "9d62681509ebe62e641a3e8530468a6e"
server_key_version: "encryptionkey@0"
I20260925 07:31:11.893364 19201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790321392278483-9480-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790321392278483-9480-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790321392278483-9480-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:31:11.896482 19201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:31:11.896732 19201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:31:11.896869 19201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:31:11.900612 19201 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.62:40125
I20260925 07:31:11.900640 19267 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.62:40125 every 8 connection(s)
I20260925 07:31:11.900950 19201 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
I20260925 07:31:11.901551 19268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:31:11.903911 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 19201
I20260925 07:31:11.904006 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790321392278483-9480-0/minicluster-data/master-0/wal/instance
I20260925 07:31:11.904196 9480 external_mini_cluster.cc:1428] Setting key cd43fa1a63b2f20930fd7d9b1877500c
I20260925 07:31:11.904246 19268 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79ce6a9d28a8493aaa389dc4c10435c2: Bootstrap starting.
I20260925 07:31:11.904888 19268 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 79ce6a9d28a8493aaa389dc4c10435c2: Neither blocks nor log segments found. Creating new log.
I20260925 07:31:11.905175 19268 log.cc:824] T 00000000000000000000000000000000 P 79ce6a9d28a8493aaa389dc4c10435c2: Log is configured to *not* fsync() on all Append() calls
I20260925 07:31:11.905918 19268 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79ce6a9d28a8493aaa389dc4c10435c2: No bootstrap required, opened a new log
I20260925 07:31:11.906661 19271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:31:11.904692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48268 (local address 127.9.66.62:40125)
0925 07:31:11.904852 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0925 07:31:11.904866 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 07:31:11.904888 (+ 22us) server_negotiation.cc:407] Connection header received
0925 07:31:11.905344 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:31:11.905357 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:31:11.905395 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:31:11.905547 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:31:11.906196 (+ 649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:31:11.906199 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:31:11.906200 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 07:31:11.906204 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:31:11.906248 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 07:31:11.906260 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:31:11.906369 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 07:31:11.906398 (+ 29us) server_negotiation.cc:299] Negotiation successful
0925 07:31:11.906463 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":73,"threads_started":1}
I20260925 07:31:11.907693 19268 raft_consensus.cc:348] T 00000000000000000000000000000000 P 79ce6a9d28a8493aaa389dc4c10435c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79ce6a9d28a8493aaa389dc4c10435c2" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 40125 } }
I20260925 07:31:11.907814 19268 raft_consensus.cc:374] T 00000000000000000000000000000000 P 79ce6a9d28a8493aaa389dc4c10435c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:31:11.907835 19268 raft_consensus.cc:729] T 00000000000000000000000000000000 P 79ce6a9d28a8493aaa389dc4c10435c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79ce6a9d28a8493aaa389dc4c10435c2, State: Initialized, Role: FOLLOWER
I20260925 07:31:11.907970 19268 consensus_queue.cc:260] T 00000000000000000000000000000000 P 79ce6a9d28a8493aaa389dc4c10435c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79ce6a9d28a8493aaa389dc4c10435c2" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 40125 } }
I20260925 07:31:11.908059 19268 raft_consensus.cc:388] T 00000000000000000000000000000000 P 79ce6a9d28a8493aaa389dc4c10435c2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 07:31:11.908082 19268 raft_consensus.cc:482] T 00000000000000000000000000000000 P 79ce6a9d28a8493aaa389dc4c10435c2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 07:31:11.908145 19268 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 79ce6a9d28a8493aaa389dc4c10435c2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:31:11.908900 19268 raft_consensus.cc:504] T 00000000000000000000000000000000 P 79ce6a9d28a8493aaa389dc4c10435c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79ce6a9d28a8493aaa389dc4c10435c2" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 40125 } }
I20260925 07:31:11.909004 19268 leader_election.cc:302] T 00000000000000000000000000000000 P 79ce6a9d28a8493aaa389dc4c10435c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 79ce6a9d28a8493aaa389dc4c10435c2; no voters:
I20260925 07:31:11.909148 19268 leader_election.cc:288] T 00000000000000000000000000000000 P 79ce6a9d28a8493aaa389dc4c10435c2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 07:31:11.909189 19273 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 79ce6a9d28a8493aaa389dc4c10435c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:31:11.909325 19273 raft_consensus.cc:686] T 00000000000000000000000000000000 P 79ce6a9d28a8493aaa389dc4c10435c2 [term 1 LEADER]: Becoming Leader. State: Replica: 79ce6a9d28a8493aaa389dc4c10435c2, State: Running, Role: LEADER
I20260925 07:31:11.909401 19268 sys_catalog.cc:564] T 00000000000000000000000000000000 P 79ce6a9d28a8493aaa389dc4c10435c2 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:31:11.909415 19273 consensus_queue.cc:237] T 00000000000000000000000000000000 P 79ce6a9d28a8493aaa389dc4c10435c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79ce6a9d28a8493aaa389dc4c10435c2" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 40125 } }
I20260925 07:31:11.909914 19275 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79ce6a9d28a8493aaa389dc4c10435c2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 79ce6a9d28a8493aaa389dc4c10435c2. Latest consensus state: current_term: 1 leader_uuid: "79ce6a9d28a8493aaa389dc4c10435c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79ce6a9d28a8493aaa389dc4c10435c2" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 40125 } } }
I20260925 07:31:11.910056 19275 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79ce6a9d28a8493aaa389dc4c10435c2 [sys.catalog]: This master's current role is: LEADER
I20260925 07:31:11.909915 19274 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79ce6a9d28a8493aaa389dc4c10435c2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "79ce6a9d28a8493aaa389dc4c10435c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79ce6a9d28a8493aaa389dc4c10435c2" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 40125 } } }
I20260925 07:31:11.910197 19274 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79ce6a9d28a8493aaa389dc4c10435c2 [sys.catalog]: This master's current role is: LEADER
I20260925 07:31:11.910554 19282 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 07:31:11.910982 19282 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 07:31:11.912216 19282 catalog_manager.cc:1347] Generated new cluster ID: e178edb685894b239082b561ca92f124
I20260925 07:31:11.912262 19282 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 07:31:11.937963 19282 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 07:31:11.938145 19282 catalog_manager.cc:1504] Loading token signing keys...
I20260925 07:31:11.941861 19282 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 79ce6a9d28a8493aaa389dc4c10435c2: Generated new TSK 0
I20260925 07:31:11.942029 19282 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 07:31:11.944286 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790321392278483-9480-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790321392278483-9480-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790321392278483-9480-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790321392278483-9480-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.66.1:0
--local_ip_for_outbound_sockets=127.9.66.1
--webserver_interface=127.9.66.1
--webserver_port=0
--tserver_master_addrs=127.9.66.62:40125
--builtin_ntp_servers=127.9.66.20:35265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 07:31:12.022796 19292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:31:12.023010 19292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:31:12.023042 19292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:31:12.024179 19292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:31:12.024240 19292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:31:12.024281 19292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.66.1
I20260925 07:31:12.025625 19292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:35265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790321392278483-9480-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790321392278483-9480-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.66.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790321392278483-9480-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.66.1
--webserver_port=0
--tserver_master_addrs=127.9.66.62:40125
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.66.1
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790321392278483-9480-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:31:12.025869 19292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:31:12.026160 19292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:31:12.031761 19300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:31:12.031783 19297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:31:12.031808 19298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:31:12.032055 19292 server_base.cc:1034] running on GCE node
I20260925 07:31:12.032232 19292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:31:12.032451 19292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:31:12.033607 19292 hybrid_clock.cc:648] HybridClock initialized: now 1790321472033568 us; error 25 us; skew 500 ppm
I20260925 07:31:12.034694 19292 webserver.cc:460] Webserver started at http://127.9.66.1:42945/ using document root <none> and password file <none>
I20260925 07:31:12.034932 19292 fs_manager.cc:359] Metadata directory not provided
I20260925 07:31:12.034974 19292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:31:12.035123 19292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:31:12.036087 19292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790321392278483-9480-0/minicluster-data/ts-0/data/instance:
uuid: "f0c2544960f84b36ad143c4e0e414000"
format_stamp: "Formatted at 2026-09-25 07:31:12 on dist-test-slave-57fn"
server_key: "0c7e19220af3d17c84a26d6cfda3860b"
server_key_iv: "ed28da93fc9cbd149307700c37a8c8ed"
server_key_version: "encryptionkey@0"
I20260925 07:31:12.036422 19292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790321392278483-9480-0/minicluster-data/ts-0/wal/instance:
uuid: "f0c2544960f84b36ad143c4e0e414000"
format_stamp: "Formatted at 2026-09-25 07:31:12 on dist-test-slave-57fn"
server_key: "0c7e19220af3d17c84a26d6cfda3860b"
server_key_iv: "ed28da93fc9cbd149307700c37a8c8ed"
server_key_version: "encryptionkey@0"
I20260925 07:31:12.037755 19292 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 07:31:12.038427 19306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:31:12.038628 19292 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 07:31:12.038714 19292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790321392278483-9480-0/minicluster-data/ts-0/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790321392278483-9480-0/minicluster-data/ts-0/wal
uuid: "f0c2544960f84b36ad143c4e0e414000"
format_stamp: "Formatted at 2026-09-25 07:31:12 on dist-test-slave-57fn"
server_key: "0c7e19220af3d17c84a26d6cfda3860b"
server_key_iv: "ed28da93fc9cbd149307700c37a8c8ed"
server_key_version: "encryptionkey@0"
I20260925 07:31:12.038785 19292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790321392278483-9480-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790321392278483-9480-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790321392278483-9480-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:31:12.041906 19292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:31:12.042138 19292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:31:12.042264 19292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:31:12.042456 19292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:31:12.042723 19292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:31:12.042752 19292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:31:12.042804 19292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:31:12.042829 19292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:31:12.048200 19292 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.1:37967
I20260925 07:31:12.048251 19419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.1:37967 every 8 connection(s)
I20260925 07:31:12.048517 19292 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790321392278483-9480-0/minicluster-data/ts-0/data/info.pb
I20260925 07:31:12.048887 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 19292
I20260925 07:31:12.048980 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790321392278483-9480-0/minicluster-data/ts-0/wal/instance
I20260925 07:31:12.049177 9480 external_mini_cluster.cc:1428] Setting key 2654330820d9fb56ae884746d789ac21
I20260925 07:31:12.050127 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790321392278483-9480-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790321392278483-9480-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790321392278483-9480-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790321392278483-9480-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.66.2:0
--local_ip_for_outbound_sockets=127.9.66.2
--webserver_interface=127.9.66.2
--webserver_port=0
--tserver_master_addrs=127.9.66.62:40125
--builtin_ntp_servers=127.9.66.20:35265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260925 07:31:12.051609 19271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:31:12.049223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.1:46549 (local address 127.9.66.62:40125)
0925 07:31:12.049263 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 07:31:12.049265 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:31:12.049549 (+ 284us) server_negotiation.cc:407] Connection header received
0925 07:31:12.050093 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:31:12.050095 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:31:12.050101 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:31:12.050158 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:31:12.051397 (+ 1239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:31:12.051398 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:31:12.051398 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 07:31:12.051402 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:31:12.051417 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 07:31:12.051421 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:31:12.051469 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 07:31:12.051540 (+ 71us) server_negotiation.cc:299] Negotiation successful
0925 07:31:12.051551 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 07:31:12.051829 19422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:31:12.049231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:40125 (local address 127.9.66.1:46549)
0925 07:31:12.049423 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0925 07:31:12.049463 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 07:31:12.049986 (+ 523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:31:12.050224 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:31:12.050228 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:31:12.050237 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:31:12.051229 (+ 992us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 07:31:12.051230 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:31:12.051270 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 07:31:12.051306 (+ 36us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:31:12.051470 (+ 164us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:31:12.051473 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:31:12.051476 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 07:31:12.051551 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 07:31:12.051612 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":87,"threads_started":1}
I20260925 07:31:12.052404 19420 heartbeater.cc:347] Connected to a master server at 127.9.66.62:40125
I20260925 07:31:12.052484 19420 heartbeater.cc:464] Registering TS with master...
I20260925 07:31:12.052634 19420 heartbeater.cc:511] Master 127.9.66.62:40125 requested a full tablet report, sending...
I20260925 07:31:12.053066 19232 ts_manager.cc:194] Registered new tserver with Master: f0c2544960f84b36ad143c4e0e414000 (127.9.66.1:37967)
W20260925 07:31:12.128463 19423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:31:12.128703 19423 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:31:12.128741 19423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:31:12.129958 19423 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:31:12.130015 19423 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:31:12.130055 19423 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.66.2
I20260925 07:31:12.131381 19423 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:35265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790321392278483-9480-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790321392278483-9480-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.66.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790321392278483-9480-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.66.2
--webserver_port=0
--tserver_master_addrs=127.9.66.62:40125
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19423
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.66.2
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790321392278483-9480-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:31:12.131646 19423 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:31:12.131934 19423 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:31:12.136890 19431 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:31:12.136927 19429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:31:12.136899 19428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:31:12.136927 19423 server_base.cc:1034] running on GCE node
I20260925 07:31:12.137355 19423 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:31:12.137650 19423 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:31:12.138808 19423 hybrid_clock.cc:648] HybridClock initialized: now 1790321472138791 us; error 45 us; skew 500 ppm
I20260925 07:31:12.140081 19423 webserver.cc:460] Webserver started at http://127.9.66.2:37451/ using document root <none> and password file <none>
I20260925 07:31:12.140377 19423 fs_manager.cc:359] Metadata directory not provided
I20260925 07:31:12.140458 19423 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:31:12.140628 19423 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:31:12.141901 19423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790321392278483-9480-0/minicluster-data/ts-1/data/instance:
uuid: "0e2f18d60df44a77befbbb845ca91a61"
format_stamp: "Formatted at 2026-09-25 07:31:12 on dist-test-slave-57fn"
server_key: "70017d358442c0d57acf82716c449e8a"
server_key_iv: "c483d49f3be4583d79cdaceca346c0f1"
server_key_version: "encryptionkey@0"
I20260925 07:31:12.142282 19423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790321392278483-9480-0/minicluster-data/ts-1/wal/instance:
uuid: "0e2f18d60df44a77befbbb845ca91a61"
format_stamp: "Formatted at 2026-09-25 07:31:12 on dist-test-slave-57fn"
server_key: "70017d358442c0d57acf82716c449e8a"
server_key_iv: "c483d49f3be4583d79cdaceca346c0f1"
server_key_version: "encryptionkey@0"
I20260925 07:31:12.143538 19423 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 07:31:12.144241 19437 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:31:12.144403 19423 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 07:31:12.144502 19423 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790321392278483-9480-0/minicluster-data/ts-1/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790321392278483-9480-0/minicluster-data/ts-1/wal
uuid: "0e2f18d60df44a77befbbb845ca91a61"
format_stamp: "Formatted at 2026-09-25 07:31:12 on dist-test-slave-57fn"
server_key: "70017d358442c0d57acf82716c449e8a"
server_key_iv: "c483d49f3be4583d79cdaceca346c0f1"
server_key_version: "encryptionkey@0"
I20260925 07:31:12.144584 19423 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790321392278483-9480-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790321392278483-9480-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790321392278483-9480-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:31:12.148052 19423 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:31:12.148320 19423 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:31:12.148483 19423 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:31:12.148690 19423 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:31:12.149011 19423 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:31:12.149070 19423 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:31:12.149117 19423 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:31:12.149148 19423 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:31:12.155437 19423 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.2:45769
I20260925 07:31:12.155469 19550 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.2:45769 every 8 connection(s)
I20260925 07:31:12.155812 19423 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790321392278483-9480-0/minicluster-data/ts-1/data/info.pb
I20260925 07:31:12.158748 19271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:31:12.156587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.2:55927 (local address 127.9.66.62:40125)
0925 07:31:12.156653 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 07:31:12.156655 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:31:12.156911 (+ 256us) server_negotiation.cc:407] Connection header received
0925 07:31:12.157357 (+ 446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:31:12.157358 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:31:12.157362 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:31:12.157421 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:31:12.158550 (+ 1129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:31:12.158551 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:31:12.158552 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 07:31:12.158555 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:31:12.158568 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 07:31:12.158572 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:31:12.158623 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 07:31:12.158684 (+ 61us) server_negotiation.cc:299] Negotiation successful
0925 07:31:12.158696 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 07:31:12.158951 19553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:31:12.156574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:40125 (local address 127.9.66.2:55927)
0925 07:31:12.156752 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0925 07:31:12.156791 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 07:31:12.157275 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:31:12.157472 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:31:12.157475 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:31:12.157483 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:31:12.158405 (+ 922us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 07:31:12.158406 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:31:12.158432 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 07:31:12.158458 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:31:12.158626 (+ 168us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:31:12.158628 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:31:12.158630 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 07:31:12.158681 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 07:31:12.158724 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":84,"threads_started":1}
I20260925 07:31:12.159384 19551 heartbeater.cc:347] Connected to a master server at 127.9.66.62:40125
I20260925 07:31:12.159484 19551 heartbeater.cc:464] Registering TS with master...
I20260925 07:31:12.159629 19551 heartbeater.cc:511] Master 127.9.66.62:40125 requested a full tablet report, sending...
I20260925 07:31:12.159977 19232 ts_manager.cc:194] Registered new tserver with Master: 0e2f18d60df44a77befbbb845ca91a61 (127.9.66.2:45769)
I20260925 07:31:12.165098 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 19423
I20260925 07:31:12.165176 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790321392278483-9480-0/minicluster-data/ts-1/wal/instance
I20260925 07:31:12.165346 9480 external_mini_cluster.cc:1428] Setting key 5a2b571fae68eaff50e5a85b466eb4a0
I20260925 07:31:12.166270 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790321392278483-9480-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790321392278483-9480-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790321392278483-9480-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790321392278483-9480-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.66.3:0
--local_ip_for_outbound_sockets=127.9.66.3
--webserver_interface=127.9.66.3
--webserver_port=0
--tserver_master_addrs=127.9.66.62:40125
--builtin_ntp_servers=127.9.66.20:35265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 07:31:12.244796 19554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:31:12.245015 19554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:31:12.245043 19554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:31:12.246208 19554 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:31:12.246248 19554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:31:12.246274 19554 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.66.3
I20260925 07:31:12.247643 19554 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:35265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790321392278483-9480-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790321392278483-9480-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.66.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790321392278483-9480-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.66.3
--webserver_port=0
--tserver_master_addrs=127.9.66.62:40125
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19554
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.66.3
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790321392278483-9480-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:31:12.247815 19554 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:31:12.248036 19554 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:31:12.252909 19560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:31:12.252933 19562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:31:12.252933 19559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:31:12.252985 19554 server_base.cc:1034] running on GCE node
I20260925 07:31:12.253350 19554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:31:12.253602 19554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:31:12.254736 19554 hybrid_clock.cc:648] HybridClock initialized: now 1790321472254727 us; error 27 us; skew 500 ppm
I20260925 07:31:12.255831 19554 webserver.cc:460] Webserver started at http://127.9.66.3:34327/ using document root <none> and password file <none>
I20260925 07:31:12.256106 19554 fs_manager.cc:359] Metadata directory not provided
I20260925 07:31:12.256150 19554 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:31:12.256319 19554 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:31:12.257361 19554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790321392278483-9480-0/minicluster-data/ts-2/data/instance:
uuid: "3d1bee7ad17f42a8a3dd19f6de37e7bd"
format_stamp: "Formatted at 2026-09-25 07:31:12 on dist-test-slave-57fn"
server_key: "1adbc1b731c04627faebf41e898fbf38"
server_key_iv: "41bdd06b932bd6c9a19cd12b0027be87"
server_key_version: "encryptionkey@0"
I20260925 07:31:12.257745 19554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790321392278483-9480-0/minicluster-data/ts-2/wal/instance:
uuid: "3d1bee7ad17f42a8a3dd19f6de37e7bd"
format_stamp: "Formatted at 2026-09-25 07:31:12 on dist-test-slave-57fn"
server_key: "1adbc1b731c04627faebf41e898fbf38"
server_key_iv: "41bdd06b932bd6c9a19cd12b0027be87"
server_key_version: "encryptionkey@0"
I20260925 07:31:12.258994 19554 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:31:12.259656 19568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:31:12.259824 19554 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 07:31:12.259908 19554 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790321392278483-9480-0/minicluster-data/ts-2/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790321392278483-9480-0/minicluster-data/ts-2/wal
uuid: "3d1bee7ad17f42a8a3dd19f6de37e7bd"
format_stamp: "Formatted at 2026-09-25 07:31:12 on dist-test-slave-57fn"
server_key: "1adbc1b731c04627faebf41e898fbf38"
server_key_iv: "41bdd06b932bd6c9a19cd12b0027be87"
server_key_version: "encryptionkey@0"
I20260925 07:31:12.259977 19554 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790321392278483-9480-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790321392278483-9480-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790321392278483-9480-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:31:12.263280 19554 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:31:12.263525 19554 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:31:12.263667 19554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:31:12.263870 19554 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:31:12.264155 19554 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:31:12.264186 19554 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:31:12.264240 19554 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:31:12.264268 19554 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:31:12.269806 19554 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.3:35499
I20260925 07:31:12.269853 19681 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.3:35499 every 8 connection(s)
I20260925 07:31:12.270176 19554 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790321392278483-9480-0/minicluster-data/ts-2/data/info.pb
I20260925 07:31:12.270486 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 19554
I20260925 07:31:12.270594 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790321392278483-9480-0/minicluster-data/ts-2/wal/instance
I20260925 07:31:12.270792 9480 external_mini_cluster.cc:1428] Setting key 30f1eb9d1bea6c0dd0c1de34a3a59512
I20260925 07:31:12.273116 19271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:31:12.270905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.3:50129 (local address 127.9.66.62:40125)
0925 07:31:12.270965 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 07:31:12.270967 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:31:12.271140 (+ 173us) server_negotiation.cc:407] Connection header received
0925 07:31:12.271608 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:31:12.271609 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:31:12.271615 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:31:12.271659 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:31:12.272815 (+ 1156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:31:12.272817 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:31:12.272817 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 07:31:12.272840 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:31:12.272859 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 07:31:12.272864 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:31:12.272927 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 07:31:12.273021 (+ 94us) server_negotiation.cc:299] Negotiation successful
0925 07:31:12.273036 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 07:31:12.273329 19684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:31:12.270862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:40125 (local address 127.9.66.3:50129)
0925 07:31:12.271025 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 07:31:12.271064 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 07:31:12.271498 (+ 434us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:31:12.271739 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:31:12.271743 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:31:12.271750 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:31:12.272693 (+ 943us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 07:31:12.272694 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:31:12.272719 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 07:31:12.272745 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:31:12.272932 (+ 187us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:31:12.272936 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:31:12.272939 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 07:31:12.273033 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 07:31:12.273103 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":75,"threads_started":1}
I20260925 07:31:12.273774 19682 heartbeater.cc:347] Connected to a master server at 127.9.66.62:40125
I20260925 07:31:12.273871 19682 heartbeater.cc:464] Registering TS with master...
I20260925 07:31:12.273996 19682 heartbeater.cc:511] Master 127.9.66.62:40125 requested a full tablet report, sending...
I20260925 07:31:12.274333 19232 ts_manager.cc:194] Registered new tserver with Master: 3d1bee7ad17f42a8a3dd19f6de37e7bd (127.9.66.3:35499)
I20260925 07:31:12.282302 9480 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 07:31:12.284209 19271 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:31:12.283661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48278 (local address 127.9.66.62:40125)
0925 07:31:12.283890 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0925 07:31:12.283892 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:31:12.283900 (+ 8us) server_negotiation.cc:407] Connection header received
0925 07:31:12.283940 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:31:12.283941 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:31:12.283944 (+ 3us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:31:12.283992 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:31:12.284135 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48278: BlockingRecv error: recv got EOF from 127.0.0.1:48278 (error 108)
Metrics: {"server-negotiator.queue_time_us":205}
I20260925 07:31:12.284338 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 19292
I20260925 07:31:12.288795 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 19423
I20260925 07:31:12.292987 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 19554
I20260925 07:31:12.297022 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 19201
2026-09-25T07:31:12Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (522 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790321392278483-9480-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 07:31:12 dist-test-slave-57fn krb5kdc[19692](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 07:31:12 dist-test-slave-57fn krb5kdc[19692](info): set up 2 sockets
Sep 25 07:31:12 dist-test-slave-57fn krb5kdc[19692](info): commencing operation
krb5kdc: starting...
W20260925 07:31:13.333840 9480 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790321392278483-9480-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790321392278483-9480-0/krb5kdc/test-user.keytab.
Sep 25 07:31:13 dist-test-slave-57fn krb5kdc[19692](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321473, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T07:31:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T07:31:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.66.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.66.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.66.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab.
Entry for principal kudu/127.9.66.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab.
I20260925 07:31:13.444141 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790321392278483-9480-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790321392278483-9480-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790321392278483-9480-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.66.62:37935
--webserver_interface=127.9.66.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab
--principal=kudu/127.9.66.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:44389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.66.62:37935
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:31:13.522662 19705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:31:13.522894 19705 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:31:13.522933 19705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:31:13.524057 19705 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:31:13.524127 19705 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:31:13.524150 19705 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:31:13.524180 19705 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:31:13.524210 19705 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:31:13.525725 19705 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:44389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790321392278483-9480-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790321392278483-9480-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.66.62:37935
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab
--principal=kudu/127.9.66.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.66.62:37935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.66.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19705
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790321392278483-9480-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:31:13.525976 19705 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:31:13.526197 19705 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:31:13.530864 19711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:31:13.530854 19710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:31:13.531008 19705 server_base.cc:1034] running on GCE node
W20260925 07:31:13.530856 19713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:31:13.531292 19705 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:31:13.531580 19705 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:31:13.532724 19705 hybrid_clock.cc:648] HybridClock initialized: now 1790321473532709 us; error 41 us; skew 500 ppm
Sep 25 07:31:13 dist-test-slave-57fn krb5kdc[19692](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321473, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:31:13.535049 19705 init.cc:377] Logged in from keytab as kudu/127.9.66.62@KRBTEST.COM (short username kudu)
I20260925 07:31:13.535488 19705 webserver.cc:460] Webserver started at http://127.9.66.62:44837/ using document root <none> and password file <none>
I20260925 07:31:13.535662 19705 fs_manager.cc:359] Metadata directory not provided
I20260925 07:31:13.535702 19705 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:31:13.535787 19705 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:31:13.536746 19705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790321392278483-9480-0/minicluster-data/master-0/data/instance:
uuid: "5e114c4aec46434f91955c8011213862"
format_stamp: "Formatted at 2026-09-25 07:31:13 on dist-test-slave-57fn"
server_key: "80e81924feee001f6c0bbf10c30b4f11"
server_key_iv: "6355765e6ec9c3caa7d5eadc450de465"
server_key_version: "encryptionkey@0"
I20260925 07:31:13.537061 19705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790321392278483-9480-0/minicluster-data/master-0/wal/instance:
uuid: "5e114c4aec46434f91955c8011213862"
format_stamp: "Formatted at 2026-09-25 07:31:13 on dist-test-slave-57fn"
server_key: "80e81924feee001f6c0bbf10c30b4f11"
server_key_iv: "6355765e6ec9c3caa7d5eadc450de465"
server_key_version: "encryptionkey@0"
I20260925 07:31:13.538419 19705 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 07:31:13.539069 19720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:31:13.539247 19705 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 07:31:13.539325 19705 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790321392278483-9480-0/minicluster-data/master-0/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790321392278483-9480-0/minicluster-data/master-0/wal
uuid: "5e114c4aec46434f91955c8011213862"
format_stamp: "Formatted at 2026-09-25 07:31:13 on dist-test-slave-57fn"
server_key: "80e81924feee001f6c0bbf10c30b4f11"
server_key_iv: "6355765e6ec9c3caa7d5eadc450de465"
server_key_version: "encryptionkey@0"
I20260925 07:31:13.539373 19705 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790321392278483-9480-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790321392278483-9480-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790321392278483-9480-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:31:13.551523 19705 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:31:13.553260 19705 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:31:13.553368 19705 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:31:13.557265 19705 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.62:37935
I20260925 07:31:13.557380 19772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.62:37935 every 8 connection(s)
I20260925 07:31:13.557729 19705 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
I20260925 07:31:13.558178 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 19705
I20260925 07:31:13.558261 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790321392278483-9480-0/minicluster-data/master-0/wal/instance
I20260925 07:31:13.558466 9480 external_mini_cluster.cc:1428] Setting key aac2330ed4c42a354621953ae921653b
I20260925 07:31:13.558960 19773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:31:13.562212 19773 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e114c4aec46434f91955c8011213862: Bootstrap starting.
I20260925 07:31:13.562894 19773 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5e114c4aec46434f91955c8011213862: Neither blocks nor log segments found. Creating new log.
I20260925 07:31:13.563197 19773 log.cc:824] T 00000000000000000000000000000000 P 5e114c4aec46434f91955c8011213862: Log is configured to *not* fsync() on all Append() calls
I20260925 07:31:13.564016 19773 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e114c4aec46434f91955c8011213862: No bootstrap required, opened a new log
Sep 25 07:31:13 dist-test-slave-57fn krb5kdc[19692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321473, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.66.62@KRBTEST.COM
I20260925 07:31:13.565887 19773 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e114c4aec46434f91955c8011213862 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e114c4aec46434f91955c8011213862" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 37935 } }
I20260925 07:31:13.566056 19773 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5e114c4aec46434f91955c8011213862 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:31:13.566095 19773 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e114c4aec46434f91955c8011213862 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e114c4aec46434f91955c8011213862, State: Initialized, Role: FOLLOWER
I20260925 07:31:13.566224 19773 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e114c4aec46434f91955c8011213862 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e114c4aec46434f91955c8011213862" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 37935 } }
I20260925 07:31:13.566330 19773 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5e114c4aec46434f91955c8011213862 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 07:31:13.566375 19773 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5e114c4aec46434f91955c8011213862 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 07:31:13.566427 19773 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e114c4aec46434f91955c8011213862 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:31:13.567139 19773 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5e114c4aec46434f91955c8011213862 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e114c4aec46434f91955c8011213862" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 37935 } }
I20260925 07:31:13.567270 19773 leader_election.cc:302] T 00000000000000000000000000000000 P 5e114c4aec46434f91955c8011213862 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5e114c4aec46434f91955c8011213862; no voters:
I20260925 07:31:13.567451 19773 leader_election.cc:288] T 00000000000000000000000000000000 P 5e114c4aec46434f91955c8011213862 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 07:31:13.567505 19778 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5e114c4aec46434f91955c8011213862 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:31:13.567711 19778 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5e114c4aec46434f91955c8011213862 [term 1 LEADER]: Becoming Leader. State: Replica: 5e114c4aec46434f91955c8011213862, State: Running, Role: LEADER
I20260925 07:31:13.567726 19773 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e114c4aec46434f91955c8011213862 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:31:13.567806 19778 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5e114c4aec46434f91955c8011213862 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e114c4aec46434f91955c8011213862" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 37935 } }
I20260925 07:31:13.568272 19778 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e114c4aec46434f91955c8011213862 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e114c4aec46434f91955c8011213862. Latest consensus state: current_term: 1 leader_uuid: "5e114c4aec46434f91955c8011213862" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e114c4aec46434f91955c8011213862" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 37935 } } }
I20260925 07:31:13.568284 19779 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e114c4aec46434f91955c8011213862 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5e114c4aec46434f91955c8011213862" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e114c4aec46434f91955c8011213862" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 37935 } } }
I20260925 07:31:13.568449 19778 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e114c4aec46434f91955c8011213862 [sys.catalog]: This master's current role is: LEADER
I20260925 07:31:13.568459 19779 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e114c4aec46434f91955c8011213862 [sys.catalog]: This master's current role is: LEADER
I20260925 07:31:13.568701 19787 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 07:31:13.569233 19787 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 07:31:13.570058 19776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:31:13.559002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43988 (local address 127.9.66.62:37935)
0925 07:31:13.559168 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 07:31:13.559177 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 07:31:13.559208 (+ 31us) server_negotiation.cc:407] Connection header received
0925 07:31:13.559745 (+ 537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:31:13.559754 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:31:13.559817 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:31:13.559969 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:31:13.560536 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:31:13.561916 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:31:13.562674 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:31:13.563297 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:31:13.563411 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:31:13.565786 (+ 2375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:31:13.565790 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:31:13.565791 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:31:13.565801 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:31:13.567776 (+ 1975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:31:13.568189 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:31:13.568191 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:31:13.568192 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:31:13.568292 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:31:13.568516 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:31:13.568517 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:31:13.568517 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:31:13.568832 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:31:13.569431 (+ 599us) server_negotiation.cc:1035] Waiting for connection context
0925 07:31:13.569499 (+ 68us) server_negotiation.cc:299] Negotiation successful
0925 07:31:13.569704 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":88,"threads_started":1}
I20260925 07:31:13.570776 19787 catalog_manager.cc:1347] Generated new cluster ID: a381e80e55c44e59b877812c70da2a7c
I20260925 07:31:13.570825 19787 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 07:31:13.583206 19787 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 07:31:13.583740 19787 catalog_manager.cc:1504] Loading token signing keys...
I20260925 07:31:13.594897 19787 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5e114c4aec46434f91955c8011213862: Generated new TSK 0
I20260925 07:31:13.595100 19787 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.66.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.66.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.66.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.1.keytab.
Entry for principal kudu/127.9.66.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.1.keytab.
I20260925 07:31:13.620919 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790321392278483-9480-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790321392278483-9480-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790321392278483-9480-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790321392278483-9480-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.66.1:0
--local_ip_for_outbound_sockets=127.9.66.1
--webserver_interface=127.9.66.1
--webserver_port=0
--tserver_master_addrs=127.9.66.62:37935
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.1.keytab
--principal=kudu/127.9.66.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:44389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:31:13.701210 19799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:31:13.701440 19799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:31:13.701463 19799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:31:13.702643 19799 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:31:13.702692 19799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:31:13.702718 19799 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.66.1
I20260925 07:31:13.704048 19799 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:44389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790321392278483-9480-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790321392278483-9480-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.1.keytab
--principal=kudu/127.9.66.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.66.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790321392278483-9480-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.66.1
--webserver_port=0
--tserver_master_addrs=127.9.66.62:37935
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19799
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.66.1
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790321392278483-9480-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:31:13.704224 19799 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:31:13.704404 19799 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:31:13.709438 19807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:31:13.709438 19804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:31:13.709762 19805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:31:13.709775 19799 server_base.cc:1034] running on GCE node
I20260925 07:31:13.709995 19799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:31:13.710239 19799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:31:13.711407 19799 hybrid_clock.cc:648] HybridClock initialized: now 1790321473711408 us; error 38 us; skew 500 ppm
Sep 25 07:31:13 dist-test-slave-57fn krb5kdc[19692](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321473, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:31:13.713770 19799 init.cc:377] Logged in from keytab as kudu/127.9.66.1@KRBTEST.COM (short username kudu)
I20260925 07:31:13.714210 19799 webserver.cc:460] Webserver started at http://127.9.66.1:35533/ using document root <none> and password file <none>
I20260925 07:31:13.714440 19799 fs_manager.cc:359] Metadata directory not provided
I20260925 07:31:13.714483 19799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:31:13.714623 19799 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:31:13.715646 19799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790321392278483-9480-0/minicluster-data/ts-0/data/instance:
uuid: "1ec38dfd5a7849f1b6952a9415d83084"
format_stamp: "Formatted at 2026-09-25 07:31:13 on dist-test-slave-57fn"
server_key: "a297d2c33e5a86220e5c3eb3a6bf31a1"
server_key_iv: "6bbf347949bc52f6e0d7af46e1f466bf"
server_key_version: "encryptionkey@0"
I20260925 07:31:13.715993 19799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790321392278483-9480-0/minicluster-data/ts-0/wal/instance:
uuid: "1ec38dfd5a7849f1b6952a9415d83084"
format_stamp: "Formatted at 2026-09-25 07:31:13 on dist-test-slave-57fn"
server_key: "a297d2c33e5a86220e5c3eb3a6bf31a1"
server_key_iv: "6bbf347949bc52f6e0d7af46e1f466bf"
server_key_version: "encryptionkey@0"
I20260925 07:31:13.717206 19799 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 07:31:13.717984 19814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:31:13.718168 19799 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 07:31:13.718247 19799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790321392278483-9480-0/minicluster-data/ts-0/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790321392278483-9480-0/minicluster-data/ts-0/wal
uuid: "1ec38dfd5a7849f1b6952a9415d83084"
format_stamp: "Formatted at 2026-09-25 07:31:13 on dist-test-slave-57fn"
server_key: "a297d2c33e5a86220e5c3eb3a6bf31a1"
server_key_iv: "6bbf347949bc52f6e0d7af46e1f466bf"
server_key_version: "encryptionkey@0"
I20260925 07:31:13.718298 19799 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790321392278483-9480-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790321392278483-9480-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790321392278483-9480-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:31:13.744045 19799 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:31:13.746266 19799 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:31:13.746380 19799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:31:13.746557 19799 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:31:13.746866 19799 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:31:13.746902 19799 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:31:13.746929 19799 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:31:13.746976 19799 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:31:13.752499 19799 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.1:41257
I20260925 07:31:13.752557 19927 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.1:41257 every 8 connection(s)
I20260925 07:31:13.752890 19799 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790321392278483-9480-0/minicluster-data/ts-0/data/info.pb
I20260925 07:31:13.755627 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 19799
I20260925 07:31:13.755708 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790321392278483-9480-0/minicluster-data/ts-0/wal/instance
I20260925 07:31:13.755894 9480 external_mini_cluster.cc:1428] Setting key 88bdf8e91470ac08247614998c951b8b
Sep 25 07:31:13 dist-test-slave-57fn krb5kdc[19692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321473, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.1@KRBTEST.COM for kudu/127.9.66.62@KRBTEST.COM
I20260925 07:31:13.762786 19776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:31:13.753632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.1:54453 (local address 127.9.66.62:37935)
0925 07:31:13.753683 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 07:31:13.753685 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:31:13.753917 (+ 232us) server_negotiation.cc:407] Connection header received
0925 07:31:13.754405 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:31:13.754406 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:31:13.754417 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:31:13.754475 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:31:13.755353 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:31:13.755974 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:31:13.757459 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:31:13.757701 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:31:13.757753 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:31:13.759943 (+ 2190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:31:13.759944 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:31:13.759945 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:31:13.759953 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:31:13.761373 (+ 1420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:31:13.761831 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:31:13.761832 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:31:13.761833 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:31:13.761894 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:31:13.762144 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:31:13.762145 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:31:13.762145 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:31:13.762259 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:31:13.762337 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 07:31:13.762608 (+ 271us) server_negotiation.cc:299] Negotiation successful
0925 07:31:13.762698 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260925 07:31:13.762953 19930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:31:13.753636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:37935 (local address 127.9.66.1:54453)
0925 07:31:13.753794 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 07:31:13.753827 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 07:31:13.754296 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:31:13.754544 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:31:13.754547 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:31:13.755090 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:31:13.755274 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:31:13.755278 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:31:13.756071 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:31:13.756072 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:31:13.757369 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:31:13.757372 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:31:13.757780 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:31:13.757782 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:31:13.757931 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:31:13.758464 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:31:13.758472 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:31:13.759845 (+ 1373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:31:13.761480 (+ 1635us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:31:13.761486 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:31:13.761488 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:31:13.761740 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:31:13.761965 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:31:13.761965 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:31:13.761966 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:31:13.762075 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:31:13.762328 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:31:13.762334 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:31:13.762470 (+ 136us) client_negotiation.cc:769] Sending connection context
0925 07:31:13.762589 (+ 119us) client_negotiation.cc:240] Negotiation successful
0925 07:31:13.762731 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":97,"threads_started":1}
I20260925 07:31:13.763463 19928 heartbeater.cc:347] Connected to a master server at 127.9.66.62:37935
I20260925 07:31:13.763563 19928 heartbeater.cc:464] Registering TS with master...
I20260925 07:31:13.763746 19928 heartbeater.cc:511] Master 127.9.66.62:37935 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.9.66.2@KRBTEST.COM; defaulting to no policy
I20260925 07:31:13.764297 19737 ts_manager.cc:194] Registered new tserver with Master: 1ec38dfd5a7849f1b6952a9415d83084 (127.9.66.1:41257)
I20260925 07:31:13.764918 19737 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.66.1@KRBTEST.COM'} at 127.9.66.1:54453
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.66.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.66.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.2.keytab.
Entry for principal kudu/127.9.66.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.2.keytab.
I20260925 07:31:13.781137 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790321392278483-9480-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790321392278483-9480-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790321392278483-9480-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790321392278483-9480-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.66.2:0
--local_ip_for_outbound_sockets=127.9.66.2
--webserver_interface=127.9.66.2
--webserver_port=0
--tserver_master_addrs=127.9.66.62:37935
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.2.keytab
--principal=kudu/127.9.66.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:44389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:31:13.859870 19933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:31:13.860054 19933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:31:13.860076 19933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:31:13.861155 19933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:31:13.861186 19933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:31:13.861212 19933 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.66.2
I20260925 07:31:13.862624 19933 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:44389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790321392278483-9480-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790321392278483-9480-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.2.keytab
--principal=kudu/127.9.66.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.66.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790321392278483-9480-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.66.2
--webserver_port=0
--tserver_master_addrs=127.9.66.62:37935
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19933
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.66.2
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790321392278483-9480-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:31:13.862809 19933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:31:13.862955 19933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:31:13.867800 19939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:31:13.867787 19933 server_base.cc:1034] running on GCE node
W20260925 07:31:13.867765 19941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:31:13.867760 19938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:31:13.868221 19933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:31:13.868484 19933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:31:13.869623 19933 hybrid_clock.cc:648] HybridClock initialized: now 1790321473869606 us; error 36 us; skew 500 ppm
Sep 25 07:31:13 dist-test-slave-57fn krb5kdc[19692](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321473, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:31:13.871922 19933 init.cc:377] Logged in from keytab as kudu/127.9.66.2@KRBTEST.COM (short username kudu)
I20260925 07:31:13.872269 19933 webserver.cc:460] Webserver started at http://127.9.66.2:38161/ using document root <none> and password file <none>
I20260925 07:31:13.872439 19933 fs_manager.cc:359] Metadata directory not provided
I20260925 07:31:13.872475 19933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:31:13.872579 19933 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:31:13.873579 19933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790321392278483-9480-0/minicluster-data/ts-1/data/instance:
uuid: "a3ed9b1736aa4be1b6a63af06849b185"
format_stamp: "Formatted at 2026-09-25 07:31:13 on dist-test-slave-57fn"
server_key: "3bec824b7cee172857fa3ac11bc9ec0c"
server_key_iv: "1e234dfb1e137e3d82babdd0415f3f92"
server_key_version: "encryptionkey@0"
I20260925 07:31:13.873885 19933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790321392278483-9480-0/minicluster-data/ts-1/wal/instance:
uuid: "a3ed9b1736aa4be1b6a63af06849b185"
format_stamp: "Formatted at 2026-09-25 07:31:13 on dist-test-slave-57fn"
server_key: "3bec824b7cee172857fa3ac11bc9ec0c"
server_key_iv: "1e234dfb1e137e3d82babdd0415f3f92"
server_key_version: "encryptionkey@0"
I20260925 07:31:13.875074 19933 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 07:31:13.875705 19948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:31:13.875845 19933 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 07:31:13.875923 19933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790321392278483-9480-0/minicluster-data/ts-1/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790321392278483-9480-0/minicluster-data/ts-1/wal
uuid: "a3ed9b1736aa4be1b6a63af06849b185"
format_stamp: "Formatted at 2026-09-25 07:31:13 on dist-test-slave-57fn"
server_key: "3bec824b7cee172857fa3ac11bc9ec0c"
server_key_iv: "1e234dfb1e137e3d82babdd0415f3f92"
server_key_version: "encryptionkey@0"
I20260925 07:31:13.875973 19933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790321392278483-9480-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790321392278483-9480-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790321392278483-9480-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:31:13.883431 19933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:31:13.885010 19933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:31:13.885124 19933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:31:13.885340 19933 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:31:13.885628 19933 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:31:13.885660 19933 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:31:13.885679 19933 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:31:13.885691 19933 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:31:13.891124 19933 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.2:35283
I20260925 07:31:13.891191 20061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.2:35283 every 8 connection(s)
I20260925 07:31:13.891470 19933 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790321392278483-9480-0/minicluster-data/ts-1/data/info.pb
I20260925 07:31:13.895407 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 19933
I20260925 07:31:13.895481 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790321392278483-9480-0/minicluster-data/ts-1/wal/instance
I20260925 07:31:13.895655 9480 external_mini_cluster.cc:1428] Setting key 11c6a86156c43d027dd010eb31e3c626
Sep 25 07:31:13 dist-test-slave-57fn krb5kdc[19692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321473, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.2@KRBTEST.COM for kudu/127.9.66.62@KRBTEST.COM
I20260925 07:31:13.900656 19776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:31:13.892248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.2:54743 (local address 127.9.66.62:37935)
0925 07:31:13.892323 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0925 07:31:13.892325 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:31:13.892510 (+ 185us) server_negotiation.cc:407] Connection header received
0925 07:31:13.892981 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:31:13.892982 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:31:13.892993 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:31:13.893037 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:31:13.893958 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:31:13.894436 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:31:13.895345 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:31:13.895624 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:31:13.895705 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:31:13.897952 (+ 2247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:31:13.897953 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:31:13.897953 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:31:13.897960 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:31:13.899263 (+ 1303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:31:13.899743 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:31:13.899744 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:31:13.899744 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:31:13.899808 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:31:13.900073 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:31:13.900073 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:31:13.900074 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:31:13.900165 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:31:13.900212 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 07:31:13.900533 (+ 321us) server_negotiation.cc:299] Negotiation successful
0925 07:31:13.900596 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 07:31:13.900965 20064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:31:13.892186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:37935 (local address 127.9.66.2:54743)
0925 07:31:13.892412 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0925 07:31:13.892439 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0925 07:31:13.892886 (+ 447us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:31:13.893107 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:31:13.893112 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:31:13.893691 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:31:13.893860 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:31:13.893864 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:31:13.894524 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:31:13.894525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:31:13.895258 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:31:13.895259 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:31:13.895714 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:31:13.895716 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:31:13.895859 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:31:13.896361 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:31:13.896369 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:31:13.897843 (+ 1474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:31:13.899343 (+ 1500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:31:13.899349 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:31:13.899351 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:31:13.899667 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:31:13.899875 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:31:13.899876 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:31:13.899876 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:31:13.900010 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:31:13.900230 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:31:13.900236 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:31:13.900396 (+ 160us) client_negotiation.cc:769] Sending connection context
0925 07:31:13.900517 (+ 121us) client_negotiation.cc:240] Negotiation successful
0925 07:31:13.900698 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":91,"threads_started":1}
I20260925 07:31:13.901494 20062 heartbeater.cc:347] Connected to a master server at 127.9.66.62:37935
I20260925 07:31:13.901623 20062 heartbeater.cc:464] Registering TS with master...
I20260925 07:31:13.901815 20062 heartbeater.cc:511] Master 127.9.66.62:37935 requested a full tablet report, sending...
I20260925 07:31:13.902182 19737 ts_manager.cc:194] Registered new tserver with Master: a3ed9b1736aa4be1b6a63af06849b185 (127.9.66.2:35283)
I20260925 07:31:13.902554 19737 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.66.2@KRBTEST.COM'} at 127.9.66.2:54743
WARNING: no policy specified for kudu/127.9.66.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.66.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.66.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.3.keytab.
Entry for principal kudu/127.9.66.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.3.keytab.
I20260925 07:31:13.920943 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790321392278483-9480-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790321392278483-9480-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790321392278483-9480-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790321392278483-9480-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.66.3:0
--local_ip_for_outbound_sockets=127.9.66.3
--webserver_interface=127.9.66.3
--webserver_port=0
--tserver_master_addrs=127.9.66.62:37935
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.3.keytab
--principal=kudu/127.9.66.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:44389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:31:13.999624 20067 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:31:13.999858 20067 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:31:13.999894 20067 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:31:14.001008 20067 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:31:14.001087 20067 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:31:14.001132 20067 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.66.3
I20260925 07:31:14.002568 20067 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:44389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790321392278483-9480-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790321392278483-9480-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.3.keytab
--principal=kudu/127.9.66.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.66.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790321392278483-9480-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.66.3
--webserver_port=0
--tserver_master_addrs=127.9.66.62:37935
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20067
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.66.3
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790321392278483-9480-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:31:14.002822 20067 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:31:14.003058 20067 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 07:31:14.007913 20067 server_base.cc:1034] running on GCE node
W20260925 07:31:14.007885 20072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:31:14.007877 20073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:31:14.007858 20075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:31:14.008283 20067 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:31:14.008569 20067 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:31:14.009730 20067 hybrid_clock.cc:648] HybridClock initialized: now 1790321474009712 us; error 36 us; skew 500 ppm
Sep 25 07:31:14 dist-test-slave-57fn krb5kdc[19692](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321474, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:31:14.020922 20067 init.cc:377] Logged in from keytab as kudu/127.9.66.3@KRBTEST.COM (short username kudu)
I20260925 07:31:14.021332 20067 webserver.cc:460] Webserver started at http://127.9.66.3:35937/ using document root <none> and password file <none>
I20260925 07:31:14.021595 20067 fs_manager.cc:359] Metadata directory not provided
I20260925 07:31:14.021679 20067 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:31:14.021792 20067 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:31:14.022833 20067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790321392278483-9480-0/minicluster-data/ts-2/data/instance:
uuid: "968e88059c20444d8f4bbce8916e4c8c"
format_stamp: "Formatted at 2026-09-25 07:31:14 on dist-test-slave-57fn"
server_key: "9db058cd3071413053b0fa4f3267e1e6"
server_key_iv: "b2aa19a153589493c182c531a24d209c"
server_key_version: "encryptionkey@0"
I20260925 07:31:14.023171 20067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790321392278483-9480-0/minicluster-data/ts-2/wal/instance:
uuid: "968e88059c20444d8f4bbce8916e4c8c"
format_stamp: "Formatted at 2026-09-25 07:31:14 on dist-test-slave-57fn"
server_key: "9db058cd3071413053b0fa4f3267e1e6"
server_key_iv: "b2aa19a153589493c182c531a24d209c"
server_key_version: "encryptionkey@0"
I20260925 07:31:14.024399 20067 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 07:31:14.025067 20082 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:31:14.025214 20067 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 07:31:14.025300 20067 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790321392278483-9480-0/minicluster-data/ts-2/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790321392278483-9480-0/minicluster-data/ts-2/wal
uuid: "968e88059c20444d8f4bbce8916e4c8c"
format_stamp: "Formatted at 2026-09-25 07:31:14 on dist-test-slave-57fn"
server_key: "9db058cd3071413053b0fa4f3267e1e6"
server_key_iv: "b2aa19a153589493c182c531a24d209c"
server_key_version: "encryptionkey@0"
I20260925 07:31:14.025354 20067 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790321392278483-9480-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790321392278483-9480-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790321392278483-9480-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:31:14.036502 20067 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:31:14.038254 20067 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:31:14.038398 20067 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:31:14.038612 20067 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:31:14.038910 20067 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:31:14.038949 20067 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:31:14.038967 20067 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:31:14.039005 20067 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:31:14.044866 20067 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.3:34475
I20260925 07:31:14.044921 20195 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.3:34475 every 8 connection(s)
I20260925 07:31:14.045224 20067 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790321392278483-9480-0/minicluster-data/ts-2/data/info.pb
Sep 25 07:31:14 dist-test-slave-57fn krb5kdc[19692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321474, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.3@KRBTEST.COM for kudu/127.9.66.62@KRBTEST.COM
I20260925 07:31:14.054263 19776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:31:14.045946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.3:46293 (local address 127.9.66.62:37935)
0925 07:31:14.046021 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0925 07:31:14.046024 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:31:14.046198 (+ 174us) server_negotiation.cc:407] Connection header received
0925 07:31:14.046703 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:31:14.046704 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:31:14.046716 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:31:14.046763 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:31:14.047664 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:31:14.048155 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:31:14.049115 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:31:14.049322 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:31:14.049405 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:31:14.051463 (+ 2058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:31:14.051465 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:31:14.051466 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:31:14.051473 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:31:14.052853 (+ 1380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:31:14.053275 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:31:14.053277 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:31:14.053277 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:31:14.053334 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:31:14.053629 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:31:14.053630 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:31:14.053630 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:31:14.053739 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:31:14.053811 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 07:31:14.054081 (+ 270us) server_negotiation.cc:299] Negotiation successful
0925 07:31:14.054177 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 07:31:14.054430 20198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:31:14.045916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:37935 (local address 127.9.66.3:46293)
0925 07:31:14.046090 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 07:31:14.046121 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0925 07:31:14.046599 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:31:14.046847 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:31:14.046850 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:31:14.047382 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:31:14.047559 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:31:14.047563 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:31:14.048253 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:31:14.048254 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:31:14.049019 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:31:14.049029 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:31:14.049413 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:31:14.049414 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:31:14.049607 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:31:14.050113 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:31:14.050121 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:31:14.051371 (+ 1250us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:31:14.052967 (+ 1596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:31:14.052973 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:31:14.052975 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:31:14.053186 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:31:14.053414 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:31:14.053414 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:31:14.053415 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:31:14.053570 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:31:14.053814 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:31:14.053820 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:31:14.053946 (+ 126us) client_negotiation.cc:769] Sending connection context
0925 07:31:14.054059 (+ 113us) client_negotiation.cc:240] Negotiation successful
0925 07:31:14.054201 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":75,"threads_started":1}
I20260925 07:31:14.054895 20196 heartbeater.cc:347] Connected to a master server at 127.9.66.62:37935
I20260925 07:31:14.054986 20196 heartbeater.cc:464] Registering TS with master...
I20260925 07:31:14.055157 20196 heartbeater.cc:511] Master 127.9.66.62:37935 requested a full tablet report, sending...
I20260925 07:31:14.055465 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 20067
I20260925 07:31:14.055558 19737 ts_manager.cc:194] Registered new tserver with Master: 968e88059c20444d8f4bbce8916e4c8c (127.9.66.3:34475)
I20260925 07:31:14.055615 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790321392278483-9480-0/minicluster-data/ts-2/wal/instance
I20260925 07:31:14.055816 9480 external_mini_cluster.cc:1428] Setting key b79a72e71a5b6b1a799ad065184dcbcc
I20260925 07:31:14.056077 19737 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.66.3@KRBTEST.COM'} at 127.9.66.3:46293
I20260925 07:31:14.056699 9480 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 07:31:14.063071 19776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:31:14.058008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43992 (local address 127.9.66.62:37935)
0925 07:31:14.058066 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 07:31:14.058068 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:31:14.058141 (+ 73us) server_negotiation.cc:407] Connection header received
0925 07:31:14.058194 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:31:14.058195 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:31:14.058202 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:31:14.058252 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:31:14.058751 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:31:14.059179 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:31:14.059832 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:31:14.059993 (+ 161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:31:14.060040 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:31:14.061119 (+ 1079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:31:14.061120 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:31:14.061120 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:31:14.061125 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:31:14.062234 (+ 1109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:31:14.062518 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:31:14.062519 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:31:14.062519 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:31:14.062560 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:31:14.062707 (+ 147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:31:14.062708 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:31:14.062708 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:31:14.062828 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:31:14.062898 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 07:31:14.062973 (+ 75us) server_negotiation.cc:299] Negotiation successful
0925 07:31:14.063020 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 07:31:14.064370 19737 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43992:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 07:31:14.066026 19737 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 07:31:14.072288 20207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:31:14.068122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41730 (local address 127.9.66.3:34475)
0925 07:31:14.068534 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0925 07:31:14.068540 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 07:31:14.068549 (+ 9us) server_negotiation.cc:407] Connection header received
0925 07:31:14.068584 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:31:14.068585 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:31:14.068599 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:31:14.068678 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:31:14.069023 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:31:14.070155 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:31:14.071395 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:31:14.071863 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:31:14.071946 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:31:14.072001 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 07:31:14.072164 (+ 163us) server_negotiation.cc:299] Negotiation successful
0925 07:31:14.072214 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":75,"threads_started":1}
I20260925 07:31:14.072280 20204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:31:14.067960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.3:34475 (local address 127.0.0.1:41730)
0925 07:31:14.068087 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0925 07:31:14.068154 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0925 07:31:14.068219 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:31:14.068880 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:31:14.068881 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:31:14.068882 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:31:14.068948 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:31:14.068949 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:31:14.070245 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:31:14.070246 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:31:14.071336 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:31:14.071338 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:31:14.071958 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:31:14.071959 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:31:14.072051 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:31:14.072063 (+ 12us) client_negotiation.cc:769] Sending connection context
0925 07:31:14.072152 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 07:31:14.072214 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":84,"threads_started":1}
I20260925 07:31:14.072379 20205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:31:14.067960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.1:41257 (local address 127.0.0.1:44220)
0925 07:31:14.068260 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0925 07:31:14.068271 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 07:31:14.068319 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:31:14.068588 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:31:14.068590 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:31:14.068592 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:31:14.068652 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:31:14.068653 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:31:14.069996 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:31:14.069997 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:31:14.071246 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:31:14.071249 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:31:14.072119 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:31:14.072120 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:31:14.072189 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:31:14.072193 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 07:31:14.072255 (+ 62us) client_negotiation.cc:240] Negotiation successful
0925 07:31:14.072302 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"mutex_wait_us":36,"thread_start_us":67,"threads_started":1}
I20260925 07:31:14.072474 20209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:31:14.068372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60406 (local address 127.9.66.2:35283)
0925 07:31:14.068540 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 07:31:14.068544 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 07:31:14.068553 (+ 9us) server_negotiation.cc:407] Connection header received
0925 07:31:14.068590 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:31:14.068593 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:31:14.068607 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:31:14.068765 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:31:14.068914 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:31:14.070111 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:31:14.071247 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:31:14.071957 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:31:14.072028 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:31:14.072082 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 07:31:14.072340 (+ 258us) server_negotiation.cc:299] Negotiation successful
0925 07:31:14.072381 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":84,"threads_started":1}
I20260925 07:31:14.072548 20208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:31:14.068174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.2:35283 (local address 127.0.0.1:60406)
0925 07:31:14.068371 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0925 07:31:14.068381 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 07:31:14.068417 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:31:14.068771 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:31:14.068772 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:31:14.068775 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:31:14.068809 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:31:14.068810 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:31:14.070224 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:31:14.070225 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:31:14.071161 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:31:14.071164 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:31:14.072120 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:31:14.072121 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:31:14.072191 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:31:14.072195 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 07:31:14.072255 (+ 60us) client_negotiation.cc:240] Negotiation successful
0925 07:31:14.072302 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"mutex_wait_us":39,"thread_start_us":32,"threads_started":1}
I20260925 07:31:14.072768 20206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:31:14.068065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44220 (local address 127.9.66.1:41257)
0925 07:31:14.068308 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0925 07:31:14.068313 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 07:31:14.068323 (+ 10us) server_negotiation.cc:407] Connection header received
0925 07:31:14.068397 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:31:14.068399 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:31:14.068416 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:31:14.068493 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:31:14.068738 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:31:14.069896 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:31:14.071319 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:31:14.071958 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:31:14.072038 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:31:14.072084 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0925 07:31:14.072649 (+ 565us) server_negotiation.cc:299] Negotiation successful
0925 07:31:14.072698 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":103,"threads_started":1}
I20260925 07:31:14.073325 19996 tablet_service.cc:1467] Processing CreateTablet for tablet 67b6c8d5c23d456cbcbfe76baf494ec0 (DEFAULT_TABLE table=test-table [id=43adb24784d9421ca40e06ed36745347]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:31:14.073342 20130 tablet_service.cc:1467] Processing CreateTablet for tablet 67b6c8d5c23d456cbcbfe76baf494ec0 (DEFAULT_TABLE table=test-table [id=43adb24784d9421ca40e06ed36745347]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:31:14.073644 20130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67b6c8d5c23d456cbcbfe76baf494ec0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:31:14.073647 19996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67b6c8d5c23d456cbcbfe76baf494ec0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:31:14.076120 20210 tablet_bootstrap.cc:492] T 67b6c8d5c23d456cbcbfe76baf494ec0 P a3ed9b1736aa4be1b6a63af06849b185: Bootstrap starting.
I20260925 07:31:14.076436 19862 tablet_service.cc:1467] Processing CreateTablet for tablet 67b6c8d5c23d456cbcbfe76baf494ec0 (DEFAULT_TABLE table=test-table [id=43adb24784d9421ca40e06ed36745347]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:31:14.076778 19862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67b6c8d5c23d456cbcbfe76baf494ec0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:31:14.076804 20210 tablet_bootstrap.cc:654] T 67b6c8d5c23d456cbcbfe76baf494ec0 P a3ed9b1736aa4be1b6a63af06849b185: Neither blocks nor log segments found. Creating new log.
I20260925 07:31:14.077330 20210 log.cc:824] T 67b6c8d5c23d456cbcbfe76baf494ec0 P a3ed9b1736aa4be1b6a63af06849b185: Log is configured to *not* fsync() on all Append() calls
I20260925 07:31:14.077755 20211 tablet_bootstrap.cc:492] T 67b6c8d5c23d456cbcbfe76baf494ec0 P 968e88059c20444d8f4bbce8916e4c8c: Bootstrap starting.
I20260925 07:31:14.078199 20210 tablet_bootstrap.cc:492] T 67b6c8d5c23d456cbcbfe76baf494ec0 P a3ed9b1736aa4be1b6a63af06849b185: No bootstrap required, opened a new log
I20260925 07:31:14.078274 20210 ts_tablet_manager.cc:1397] T 67b6c8d5c23d456cbcbfe76baf494ec0 P a3ed9b1736aa4be1b6a63af06849b185: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260925 07:31:14.078365 20211 tablet_bootstrap.cc:654] T 67b6c8d5c23d456cbcbfe76baf494ec0 P 968e88059c20444d8f4bbce8916e4c8c: Neither blocks nor log segments found. Creating new log.
I20260925 07:31:14.078836 20211 log.cc:824] T 67b6c8d5c23d456cbcbfe76baf494ec0 P 968e88059c20444d8f4bbce8916e4c8c: Log is configured to *not* fsync() on all Append() calls
I20260925 07:31:14.079703 20211 tablet_bootstrap.cc:492] T 67b6c8d5c23d456cbcbfe76baf494ec0 P 968e88059c20444d8f4bbce8916e4c8c: No bootstrap required, opened a new log
I20260925 07:31:14.079780 20211 ts_tablet_manager.cc:1397] T 67b6c8d5c23d456cbcbfe76baf494ec0 P 968e88059c20444d8f4bbce8916e4c8c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 07:31:14.080000 20214 tablet_bootstrap.cc:492] T 67b6c8d5c23d456cbcbfe76baf494ec0 P 1ec38dfd5a7849f1b6952a9415d83084: Bootstrap starting.
I20260925 07:31:14.080340 20210 raft_consensus.cc:348] T 67b6c8d5c23d456cbcbfe76baf494ec0 P a3ed9b1736aa4be1b6a63af06849b185 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ec38dfd5a7849f1b6952a9415d83084" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 41257 } } peers { permanent_uuid: "968e88059c20444d8f4bbce8916e4c8c" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 34475 } } peers { permanent_uuid: "a3ed9b1736aa4be1b6a63af06849b185" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 35283 } }
I20260925 07:31:14.080462 20210 raft_consensus.cc:374] T 67b6c8d5c23d456cbcbfe76baf494ec0 P a3ed9b1736aa4be1b6a63af06849b185 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:31:14.080510 20210 raft_consensus.cc:729] T 67b6c8d5c23d456cbcbfe76baf494ec0 P a3ed9b1736aa4be1b6a63af06849b185 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3ed9b1736aa4be1b6a63af06849b185, State: Initialized, Role: FOLLOWER
I20260925 07:31:14.080633 20210 consensus_queue.cc:260] T 67b6c8d5c23d456cbcbfe76baf494ec0 P a3ed9b1736aa4be1b6a63af06849b185 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ec38dfd5a7849f1b6952a9415d83084" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 41257 } } peers { permanent_uuid: "968e88059c20444d8f4bbce8916e4c8c" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 34475 } } peers { permanent_uuid: "a3ed9b1736aa4be1b6a63af06849b185" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 35283 } }
I20260925 07:31:14.080744 20214 tablet_bootstrap.cc:654] T 67b6c8d5c23d456cbcbfe76baf494ec0 P 1ec38dfd5a7849f1b6952a9415d83084: Neither blocks nor log segments found. Creating new log.
I20260925 07:31:14.080909 20210 ts_tablet_manager.cc:1428] T 67b6c8d5c23d456cbcbfe76baf494ec0 P a3ed9b1736aa4be1b6a63af06849b185: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 07:31:14.081048 20062 heartbeater.cc:503] Master 127.9.66.62:37935 was elected leader, sending a full tablet report...
I20260925 07:31:14.081338 20214 log.cc:824] T 67b6c8d5c23d456cbcbfe76baf494ec0 P 1ec38dfd5a7849f1b6952a9415d83084: Log is configured to *not* fsync() on all Append() calls
I20260925 07:31:14.081915 20211 raft_consensus.cc:348] T 67b6c8d5c23d456cbcbfe76baf494ec0 P 968e88059c20444d8f4bbce8916e4c8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ec38dfd5a7849f1b6952a9415d83084" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 41257 } } peers { permanent_uuid: "968e88059c20444d8f4bbce8916e4c8c" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 34475 } } peers { permanent_uuid: "a3ed9b1736aa4be1b6a63af06849b185" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 35283 } }
I20260925 07:31:14.082060 20214 tablet_bootstrap.cc:492] T 67b6c8d5c23d456cbcbfe76baf494ec0 P 1ec38dfd5a7849f1b6952a9415d83084: No bootstrap required, opened a new log
I20260925 07:31:14.082062 20211 raft_consensus.cc:374] T 67b6c8d5c23d456cbcbfe76baf494ec0 P 968e88059c20444d8f4bbce8916e4c8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:31:14.082108 20211 raft_consensus.cc:729] T 67b6c8d5c23d456cbcbfe76baf494ec0 P 968e88059c20444d8f4bbce8916e4c8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 968e88059c20444d8f4bbce8916e4c8c, State: Initialized, Role: FOLLOWER
I20260925 07:31:14.082132 20214 ts_tablet_manager.cc:1397] T 67b6c8d5c23d456cbcbfe76baf494ec0 P 1ec38dfd5a7849f1b6952a9415d83084: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260925 07:31:14.082240 20211 consensus_queue.cc:260] T 67b6c8d5c23d456cbcbfe76baf494ec0 P 968e88059c20444d8f4bbce8916e4c8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ec38dfd5a7849f1b6952a9415d83084" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 41257 } } peers { permanent_uuid: "968e88059c20444d8f4bbce8916e4c8c" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 34475 } } peers { permanent_uuid: "a3ed9b1736aa4be1b6a63af06849b185" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 35283 } }
I20260925 07:31:14.082495 20211 ts_tablet_manager.cc:1428] T 67b6c8d5c23d456cbcbfe76baf494ec0 P 968e88059c20444d8f4bbce8916e4c8c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 07:31:14.082532 20196 heartbeater.cc:503] Master 127.9.66.62:37935 was elected leader, sending a full tablet report...
I20260925 07:31:14.083930 20214 raft_consensus.cc:348] T 67b6c8d5c23d456cbcbfe76baf494ec0 P 1ec38dfd5a7849f1b6952a9415d83084 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ec38dfd5a7849f1b6952a9415d83084" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 41257 } } peers { permanent_uuid: "968e88059c20444d8f4bbce8916e4c8c" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 34475 } } peers { permanent_uuid: "a3ed9b1736aa4be1b6a63af06849b185" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 35283 } }
I20260925 07:31:14.084039 20214 raft_consensus.cc:374] T 67b6c8d5c23d456cbcbfe76baf494ec0 P 1ec38dfd5a7849f1b6952a9415d83084 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:31:14.084059 20214 raft_consensus.cc:729] T 67b6c8d5c23d456cbcbfe76baf494ec0 P 1ec38dfd5a7849f1b6952a9415d83084 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ec38dfd5a7849f1b6952a9415d83084, State: Initialized, Role: FOLLOWER
I20260925 07:31:14.084131 20214 consensus_queue.cc:260] T 67b6c8d5c23d456cbcbfe76baf494ec0 P 1ec38dfd5a7849f1b6952a9415d83084 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ec38dfd5a7849f1b6952a9415d83084" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 41257 } } peers { permanent_uuid: "968e88059c20444d8f4bbce8916e4c8c" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 34475 } } peers { permanent_uuid: "a3ed9b1736aa4be1b6a63af06849b185" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 35283 } }
I20260925 07:31:14.084311 20214 ts_tablet_manager.cc:1428] T 67b6c8d5c23d456cbcbfe76baf494ec0 P 1ec38dfd5a7849f1b6952a9415d83084: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260925 07:31:14.084365 19928 heartbeater.cc:503] Master 127.9.66.62:37935 was elected leader, sending a full tablet report...
I20260925 07:31:14.137952 20215 raft_consensus.cc:482] T 67b6c8d5c23d456cbcbfe76baf494ec0 P a3ed9b1736aa4be1b6a63af06849b185 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 07:31:14.138098 20215 raft_consensus.cc:504] T 67b6c8d5c23d456cbcbfe76baf494ec0 P a3ed9b1736aa4be1b6a63af06849b185 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ec38dfd5a7849f1b6952a9415d83084" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 41257 } } peers { permanent_uuid: "968e88059c20444d8f4bbce8916e4c8c" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 34475 } } peers { permanent_uuid: "a3ed9b1736aa4be1b6a63af06849b185" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 35283 } }
I20260925 07:31:14.138504 20215 leader_election.cc:288] T 67b6c8d5c23d456cbcbfe76baf494ec0 P a3ed9b1736aa4be1b6a63af06849b185 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1ec38dfd5a7849f1b6952a9415d83084 (127.9.66.1:41257), 968e88059c20444d8f4bbce8916e4c8c (127.9.66.3:34475)
I20260925 07:31:14.141714 20206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:31:14.138768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.2:58539 (local address 127.9.66.1:41257)
0925 07:31:14.138824 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 07:31:14.138826 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:31:14.138835 (+ 9us) server_negotiation.cc:407] Connection header received
0925 07:31:14.138899 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:31:14.138899 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:31:14.138909 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:31:14.138966 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:31:14.139145 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:31:14.139713 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:31:14.140908 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:31:14.141353 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:31:14.141406 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:31:14.141414 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 07:31:14.141592 (+ 178us) server_negotiation.cc:299] Negotiation successful
0925 07:31:14.141642 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 07:31:14.141752 20219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:31:14.138714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.3:34475 (local address 127.9.66.2:52115)
0925 07:31:14.139072 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0925 07:31:14.139081 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 07:31:14.139149 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:31:14.139269 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:31:14.139270 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:31:14.139271 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:31:14.139330 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:31:14.139331 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:31:14.139970 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:31:14.139971 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:31:14.140934 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:31:14.140937 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:31:14.141468 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:31:14.141469 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:31:14.141573 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:31:14.141578 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 07:31:14.141634 (+ 56us) client_negotiation.cc:240] Negotiation successful
0925 07:31:14.141667 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"mutex_wait_us":133,"thread_start_us":143,"threads_started":1}
I20260925 07:31:14.141788 20207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:31:14.138758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.2:52115 (local address 127.9.66.3:34475)
0925 07:31:14.138834 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0925 07:31:14.138837 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:31:14.139165 (+ 328us) server_negotiation.cc:407] Connection header received
0925 07:31:14.139197 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:31:14.139198 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:31:14.139208 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:31:14.139250 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:31:14.139393 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:31:14.139897 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:31:14.141013 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:31:14.141366 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:31:14.141455 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:31:14.141464 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 07:31:14.141650 (+ 186us) server_negotiation.cc:299] Negotiation successful
0925 07:31:14.141708 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 07:31:14.141726 20064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:31:14.138714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.1:41257 (local address 127.9.66.2:58539)
0925 07:31:14.138775 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0925 07:31:14.138786 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 07:31:14.138827 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:31:14.138987 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:31:14.138987 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:31:14.138989 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:31:14.139060 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:31:14.139061 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:31:14.139794 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:31:14.139795 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:31:14.140835 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:31:14.140837 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:31:14.141434 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:31:14.141435 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:31:14.141533 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:31:14.141537 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 07:31:14.141585 (+ 48us) client_negotiation.cc:240] Negotiation successful
0925 07:31:14.141648 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
W20260925 07:31:14.142016 20063 tablet.cc:2369] T 67b6c8d5c23d456cbcbfe76baf494ec0 P a3ed9b1736aa4be1b6a63af06849b185: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 07:31:14.142097 19882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67b6c8d5c23d456cbcbfe76baf494ec0" candidate_uuid: "a3ed9b1736aa4be1b6a63af06849b185" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ec38dfd5a7849f1b6952a9415d83084" is_pre_election: true
I20260925 07:31:14.142184 20150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67b6c8d5c23d456cbcbfe76baf494ec0" candidate_uuid: "a3ed9b1736aa4be1b6a63af06849b185" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "968e88059c20444d8f4bbce8916e4c8c" is_pre_election: true
I20260925 07:31:14.142254 19882 raft_consensus.cc:2454] T 67b6c8d5c23d456cbcbfe76baf494ec0 P 1ec38dfd5a7849f1b6952a9415d83084 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3ed9b1736aa4be1b6a63af06849b185 in term 0.
I20260925 07:31:14.142356 20150 raft_consensus.cc:2454] T 67b6c8d5c23d456cbcbfe76baf494ec0 P 968e88059c20444d8f4bbce8916e4c8c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3ed9b1736aa4be1b6a63af06849b185 in term 0.
I20260925 07:31:14.142508 19951 leader_election.cc:302] T 67b6c8d5c23d456cbcbfe76baf494ec0 P a3ed9b1736aa4be1b6a63af06849b185 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ec38dfd5a7849f1b6952a9415d83084, a3ed9b1736aa4be1b6a63af06849b185; no voters:
I20260925 07:31:14.142659 20215 raft_consensus.cc:2781] T 67b6c8d5c23d456cbcbfe76baf494ec0 P a3ed9b1736aa4be1b6a63af06849b185 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 07:31:14.142750 20215 raft_consensus.cc:482] T 67b6c8d5c23d456cbcbfe76baf494ec0 P a3ed9b1736aa4be1b6a63af06849b185 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 07:31:14.142781 20215 raft_consensus.cc:3037] T 67b6c8d5c23d456cbcbfe76baf494ec0 P a3ed9b1736aa4be1b6a63af06849b185 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:31:14.143493 20215 raft_consensus.cc:504] T 67b6c8d5c23d456cbcbfe76baf494ec0 P a3ed9b1736aa4be1b6a63af06849b185 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ec38dfd5a7849f1b6952a9415d83084" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 41257 } } peers { permanent_uuid: "968e88059c20444d8f4bbce8916e4c8c" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 34475 } } peers { permanent_uuid: "a3ed9b1736aa4be1b6a63af06849b185" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 35283 } }
I20260925 07:31:14.143641 20215 leader_election.cc:288] T 67b6c8d5c23d456cbcbfe76baf494ec0 P a3ed9b1736aa4be1b6a63af06849b185 [CANDIDATE]: Term 1 election: Requested vote from peers 1ec38dfd5a7849f1b6952a9415d83084 (127.9.66.1:41257), 968e88059c20444d8f4bbce8916e4c8c (127.9.66.3:34475)
I20260925 07:31:14.143800 19882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67b6c8d5c23d456cbcbfe76baf494ec0" candidate_uuid: "a3ed9b1736aa4be1b6a63af06849b185" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ec38dfd5a7849f1b6952a9415d83084"
I20260925 07:31:14.143882 19882 raft_consensus.cc:3037] T 67b6c8d5c23d456cbcbfe76baf494ec0 P 1ec38dfd5a7849f1b6952a9415d83084 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:31:14.143985 20150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67b6c8d5c23d456cbcbfe76baf494ec0" candidate_uuid: "a3ed9b1736aa4be1b6a63af06849b185" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "968e88059c20444d8f4bbce8916e4c8c"
I20260925 07:31:14.144093 20150 raft_consensus.cc:3037] T 67b6c8d5c23d456cbcbfe76baf494ec0 P 968e88059c20444d8f4bbce8916e4c8c [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:31:14.144637 19882 raft_consensus.cc:2454] T 67b6c8d5c23d456cbcbfe76baf494ec0 P 1ec38dfd5a7849f1b6952a9415d83084 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3ed9b1736aa4be1b6a63af06849b185 in term 1.
I20260925 07:31:14.144793 20150 raft_consensus.cc:2454] T 67b6c8d5c23d456cbcbfe76baf494ec0 P 968e88059c20444d8f4bbce8916e4c8c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3ed9b1736aa4be1b6a63af06849b185 in term 1.
I20260925 07:31:14.144826 19951 leader_election.cc:302] T 67b6c8d5c23d456cbcbfe76baf494ec0 P a3ed9b1736aa4be1b6a63af06849b185 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ec38dfd5a7849f1b6952a9415d83084, a3ed9b1736aa4be1b6a63af06849b185; no voters:
I20260925 07:31:14.144902 20215 raft_consensus.cc:2781] T 67b6c8d5c23d456cbcbfe76baf494ec0 P a3ed9b1736aa4be1b6a63af06849b185 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:31:14.145059 20215 raft_consensus.cc:686] T 67b6c8d5c23d456cbcbfe76baf494ec0 P a3ed9b1736aa4be1b6a63af06849b185 [term 1 LEADER]: Becoming Leader. State: Replica: a3ed9b1736aa4be1b6a63af06849b185, State: Running, Role: LEADER
I20260925 07:31:14.145172 20215 consensus_queue.cc:237] T 67b6c8d5c23d456cbcbfe76baf494ec0 P a3ed9b1736aa4be1b6a63af06849b185 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ec38dfd5a7849f1b6952a9415d83084" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 41257 } } peers { permanent_uuid: "968e88059c20444d8f4bbce8916e4c8c" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 34475 } } peers { permanent_uuid: "a3ed9b1736aa4be1b6a63af06849b185" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 35283 } }
I20260925 07:31:14.145956 19737 catalog_manager.cc:5495] T 67b6c8d5c23d456cbcbfe76baf494ec0 P a3ed9b1736aa4be1b6a63af06849b185 reported cstate change: term changed from 0 to 1, leader changed from <none> to a3ed9b1736aa4be1b6a63af06849b185 (127.9.66.2). New cstate: current_term: 1 leader_uuid: "a3ed9b1736aa4be1b6a63af06849b185" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ec38dfd5a7849f1b6952a9415d83084" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 41257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "968e88059c20444d8f4bbce8916e4c8c" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 34475 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3ed9b1736aa4be1b6a63af06849b185" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 35283 } health_report { overall_health: HEALTHY } } }
I20260925 07:31:14.163017 20209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:31:14.160908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60410 (local address 127.9.66.2:35283)
0925 07:31:14.160963 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 07:31:14.160965 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:31:14.160974 (+ 9us) server_negotiation.cc:407] Connection header received
0925 07:31:14.161021 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:31:14.161021 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:31:14.161039 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:31:14.161095 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 07:31:14.161209 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:31:14.161714 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:31:14.162389 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:31:14.162555 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:31:14.162607 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:31:14.162733 (+ 126us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 07:31:14.162820 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 07:31:14.162875 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 07:31:14.162887 (+ 12us) server_negotiation.cc:299] Negotiation successful
0925 07:31:14.162902 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 07:31:14.164587 20150 raft_consensus.cc:1261] T 67b6c8d5c23d456cbcbfe76baf494ec0 P 968e88059c20444d8f4bbce8916e4c8c [term 1 FOLLOWER]: Refusing update from remote peer a3ed9b1736aa4be1b6a63af06849b185: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:31:14.164598 19882 raft_consensus.cc:1261] T 67b6c8d5c23d456cbcbfe76baf494ec0 P 1ec38dfd5a7849f1b6952a9415d83084 [term 1 FOLLOWER]: Refusing update from remote peer a3ed9b1736aa4be1b6a63af06849b185: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:31:14.164906 20220 consensus_queue.cc:1035] T 67b6c8d5c23d456cbcbfe76baf494ec0 P a3ed9b1736aa4be1b6a63af06849b185 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ec38dfd5a7849f1b6952a9415d83084" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 41257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:31:14.165017 20215 consensus_queue.cc:1035] T 67b6c8d5c23d456cbcbfe76baf494ec0 P a3ed9b1736aa4be1b6a63af06849b185 [LEADER]: Connected to new peer: Peer: permanent_uuid: "968e88059c20444d8f4bbce8916e4c8c" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 34475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:31:14.168262 20225 mvcc.cc:205] Tried to move back new op lower bound from 7333156758175592448 to 7333156758099189760. Current Snapshot: MvccSnapshot[applied={T|T < 7333156758175592448 or (T in {7333156758175592448})}]
I20260925 07:31:14.168367 20224 mvcc.cc:205] Tried to move back new op lower bound from 7333156758175592448 to 7333156758099189760. Current Snapshot: MvccSnapshot[applied={T|T < 7333156758175592448 or (T in {7333156758175592448})}]
I20260925 07:31:14.169530 19736 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43992:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 07:31:14.169626 19736 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43992:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 07:31:14.170343 19736 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5e114c4aec46434f91955c8011213862: Sending DeleteTablet for 3 replicas of tablet 67b6c8d5c23d456cbcbfe76baf494ec0
I20260925 07:31:14.170642 19862 tablet_service.cc:1514] Processing DeleteTablet for tablet 67b6c8d5c23d456cbcbfe76baf494ec0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:31:14 UTC) from {username='kudu', principal='kudu/127.9.66.62@KRBTEST.COM'} at 127.0.0.1:44220
I20260925 07:31:14.170665 20130 tablet_service.cc:1514] Processing DeleteTablet for tablet 67b6c8d5c23d456cbcbfe76baf494ec0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:31:14 UTC) from {username='kudu', principal='kudu/127.9.66.62@KRBTEST.COM'} at 127.0.0.1:41730
I20260925 07:31:14.170687 19996 tablet_service.cc:1514] Processing DeleteTablet for tablet 67b6c8d5c23d456cbcbfe76baf494ec0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:31:14 UTC) from {username='kudu', principal='kudu/127.9.66.62@KRBTEST.COM'} at 127.0.0.1:60406
I20260925 07:31:14.170882 20234 tablet_replica.cc:330] T 67b6c8d5c23d456cbcbfe76baf494ec0 P 968e88059c20444d8f4bbce8916e4c8c: stopping tablet replica
I20260925 07:31:14.170971 20234 raft_consensus.cc:2229] T 67b6c8d5c23d456cbcbfe76baf494ec0 P 968e88059c20444d8f4bbce8916e4c8c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 07:31:14.171074 20234 raft_consensus.cc:2258] T 67b6c8d5c23d456cbcbfe76baf494ec0 P 968e88059c20444d8f4bbce8916e4c8c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:31:14.171167 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 19799
I20260925 07:31:14.172679 20235 tablet_replica.cc:330] T 67b6c8d5c23d456cbcbfe76baf494ec0 P a3ed9b1736aa4be1b6a63af06849b185: stopping tablet replica
I20260925 07:31:14.172775 20235 raft_consensus.cc:2229] T 67b6c8d5c23d456cbcbfe76baf494ec0 P a3ed9b1736aa4be1b6a63af06849b185 [term 1 LEADER]: Raft consensus shutting down.
I20260925 07:31:14.172859 20235 raft_consensus.cc:2258] T 67b6c8d5c23d456cbcbfe76baf494ec0 P a3ed9b1736aa4be1b6a63af06849b185 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:31:14.173355 20235 ts_tablet_manager.cc:1905] T 67b6c8d5c23d456cbcbfe76baf494ec0 P a3ed9b1736aa4be1b6a63af06849b185: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 07:31:14.174747 20234 ts_tablet_manager.cc:1905] T 67b6c8d5c23d456cbcbfe76baf494ec0 P 968e88059c20444d8f4bbce8916e4c8c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 07:31:14.174772 20235 ts_tablet_manager.cc:1918] T 67b6c8d5c23d456cbcbfe76baf494ec0 P a3ed9b1736aa4be1b6a63af06849b185: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 07:31:14.174829 20235 log.cc:1196] T 67b6c8d5c23d456cbcbfe76baf494ec0 P a3ed9b1736aa4be1b6a63af06849b185: Deleting WAL directory at /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790321392278483-9480-0/minicluster-data/ts-1/wal/wals/67b6c8d5c23d456cbcbfe76baf494ec0
I20260925 07:31:14.175096 20235 ts_tablet_manager.cc:1939] T 67b6c8d5c23d456cbcbfe76baf494ec0 P a3ed9b1736aa4be1b6a63af06849b185: Deleting consensus metadata
I20260925 07:31:14.175467 19723 catalog_manager.cc:4841] TS a3ed9b1736aa4be1b6a63af06849b185 (127.9.66.2:35283): tablet 67b6c8d5c23d456cbcbfe76baf494ec0 (table test-table [id=43adb24784d9421ca40e06ed36745347]) successfully deleted
I20260925 07:31:14.176143 20234 ts_tablet_manager.cc:1918] T 67b6c8d5c23d456cbcbfe76baf494ec0 P 968e88059c20444d8f4bbce8916e4c8c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 07:31:14.176195 20234 log.cc:1196] T 67b6c8d5c23d456cbcbfe76baf494ec0 P 968e88059c20444d8f4bbce8916e4c8c: Deleting WAL directory at /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790321392278483-9480-0/minicluster-data/ts-2/wal/wals/67b6c8d5c23d456cbcbfe76baf494ec0
I20260925 07:31:14.176437 20234 ts_tablet_manager.cc:1939] T 67b6c8d5c23d456cbcbfe76baf494ec0 P 968e88059c20444d8f4bbce8916e4c8c: Deleting consensus metadata
I20260925 07:31:14.176784 19724 catalog_manager.cc:4841] TS 968e88059c20444d8f4bbce8916e4c8c (127.9.66.3:34475): tablet 67b6c8d5c23d456cbcbfe76baf494ec0 (table test-table [id=43adb24784d9421ca40e06ed36745347]) successfully deleted
W20260925 07:31:14.184388 19723 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.66.1:41257 (error 108)
I20260925 07:31:14.184885 20205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:31:14.184744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.1:41257 (local address 127.0.0.1:44236)
0925 07:31:14.184798 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0925 07:31:14.184840 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.1:41257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260925 07:31:14.184942 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 19933
W20260925 07:31:14.186424 19723 catalog_manager.cc:4584] TS 1ec38dfd5a7849f1b6952a9415d83084 (127.9.66.1:41257): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 67b6c8d5c23d456cbcbfe76baf494ec0: Network error: Client connection negotiation failed: client connection to 127.9.66.1:41257: connect: Connection refused (error 111)
I20260925 07:31:14.189488 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 20067
I20260925 07:31:14.193910 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 19705
2026-09-25T07:31:14Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (1897 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 07:31:14 dist-test-slave-57fn krb5kdc[20239](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 07:31:14 dist-test-slave-57fn krb5kdc[20239](info): set up 2 sockets
Sep 25 07:31:14 dist-test-slave-57fn krb5kdc[20239](info): commencing operation
krb5kdc: starting...
W20260925 07:31:15.217839 9480 mini_kdc.cc:121] Time spent starting KDC: real 1.004s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/krb5kdc/test-user.keytab.
Sep 25 07:31:15 dist-test-slave-57fn krb5kdc[20239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321475, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T07:31:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T07:31:15Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.9.66.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.9.66.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.9.66.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/krb5kdc/rangeradmin_127.9.66.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.9.66.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/krb5kdc/rangeradmin_127.9.66.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.9.66.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.9.66.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.9.66.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/krb5kdc/rangerlookup_127.9.66.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.9.66.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/krb5kdc/rangerlookup_127.9.66.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.9.66.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.66.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.66.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/krb5kdc/HTTP_127.9.66.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.9.66.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/krb5kdc/HTTP_127.9.66.20@KRBTEST.COM.keytab.
I20260925 07:31:15.370483 9480 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task1eZXJS/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/minicluster-data/postgres -l logfile start
2026-09-25 07:31:17.757 UTC [20269] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-25 07:31:17.757 UTC [20269] LOG: listening on IPv4 address "127.9.66.20", port 35737
2026-09-25 07:31:17.761 UTC [20269] LOG: listening on Unix socket "/tmp/.s.PGSQL.35737"
2026-09-25 07:31:17.795 UTC [20272] LOG: database system was shut down at 2026-09-25 07:31:16 UTC
2026-09-25 07:31:17.799 UTC [20269] LOG: database system is ready to accept connections
I20260925 07:31:18.741611 9480 mini_postgres.cc:96] Postgres bound to 35737
2026-09-25 07:31:18.747 UTC [20280] FATAL: database "slave" does not exist
127.9.66.20:35737 - accepting connections
I20260925 07:31:18.748524 9480 mini_ranger.cc:161] Starting Ranger...
I20260925 07:31:18.764117 9480 mini_ranger.cc:84] Created miniranger Postgres user
I20260925 07:31:19.218351 9480 mini_ranger.cc:87] Created ranger Postgres database
I20260925 07:31:19.218451 9480 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/minicluster-data/ranger-admin
2026-09-25 07:31:19,496 [I] DB FLAVOR :POSTGRES
2026-09-25 07:31:19,496 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 07:31:19,496 [I] Checking connection..
2026-09-25 07:31:19,496 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1eZXJS/build/release/bin/postgresql.jar:/tmp/dist-test-task1eZXJS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.66.20:35737/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-25 07:31:19,729 [I] Checking connection passed.
2026-09-25 07:31:19,729 [I] --------- Verifying version history table ---------
2026-09-25 07:31:19,730 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1eZXJS/build/release/bin/postgresql.jar:/tmp/dist-test-task1eZXJS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.66.20:35737/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 07:31:19,956 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 07:31:19,957 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1eZXJS/build/release/bin/postgresql.jar:/tmp/dist-test-task1eZXJS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.66.20:35737/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 07:31:20,205 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 07:31:20,205 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-25 07:31:20,206 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1eZXJS/build/release/bin/postgresql.jar:/tmp/dist-test-task1eZXJS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.66.20:35737/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task1eZXJS/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-25 07:31:20.439 UTC [20344] WARNING: there is no transaction in progress
2026-09-25 07:31:20,445 [I] create_dbversion_catalog.sql file imported successfully
2026-09-25 07:31:20,445 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1eZXJS/build/release/bin/postgresql.jar:/tmp/dist-test-task1eZXJS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.66.20:35737/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 07:31:20,675 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-25 07:31:20,675 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 07:31:20,675 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1eZXJS/build/release/bin/postgresql.jar:/tmp/dist-test-task1eZXJS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.66.20:35737/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-25 07:31:20,914 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1eZXJS/build/release/bin/postgresql.jar:/tmp/dist-test-task1eZXJS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.66.20:35737/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-25 07:31:21,156 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1eZXJS/build/release/bin/postgresql.jar:/tmp/dist-test-task1eZXJS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.66.20:35737/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-57fn.c.gcp-upstream.internal','N') ;"
2026-09-25 07:31:21,385 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1eZXJS/build/release/bin/postgresql.jar:/tmp/dist-test-task1eZXJS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.66.20:35737/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 07:31:21,639 [I] Table x_portal_user does not exist in database ranger
2026-09-25 07:31:21,639 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1eZXJS/build/release/bin/postgresql.jar:/tmp/dist-test-task1eZXJS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.66.20:35737/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 07:31:21,891 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-25 07:31:21,891 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1eZXJS/build/release/bin/postgresql.jar:/tmp/dist-test-task1eZXJS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.66.20:35737/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 07:31:22,141 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1eZXJS/build/release/bin/postgresql.jar:/tmp/dist-test-task1eZXJS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.66.20:35737/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 07:31:22,388 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-25 07:31:22,389 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1eZXJS/build/release/bin/postgresql.jar:/tmp/dist-test-task1eZXJS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.66.20:35737/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task1eZXJS/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-25 07:31:23.340 UTC [20452] WARNING: there is no transaction in progress
2026-09-25 07:31:23.356 UTC [20452] WARNING: there is no transaction in progress
2026-09-25 07:31:23.371 UTC [20452] WARNING: there is no transaction in progress
2026-09-25 07:31:23.385 UTC [20452] WARNING: there is no transaction in progress
2026-09-25 07:31:23.403 UTC [20452] WARNING: there is no transaction in progress
2026-09-25 07:31:23.421 UTC [20452] WARNING: there is no transaction in progress
2026-09-25 07:31:23.474 UTC [20452] WARNING: there is no transaction in progress
2026-09-25 07:31:23.480 UTC [20452] WARNING: there is no transaction in progress
2026-09-25 07:31:23.493 UTC [20452] WARNING: there is no transaction in progress
2026-09-25 07:31:23.503 UTC [20452] WARNING: there is no transaction in progress
2026-09-25 07:31:23.512 UTC [20452] WARNING: there is no transaction in progress
2026-09-25 07:31:23.526 UTC [20452] WARNING: there is no transaction in progress
2026-09-25 07:31:23.535 UTC [20452] WARNING: there is no transaction in progress
2026-09-25 07:31:23.541 UTC [20452] WARNING: there is no transaction in progress
2026-09-25 07:31:23.548 UTC [20452] WARNING: there is no transaction in progress
2026-09-25 07:31:23.555 UTC [20452] WARNING: there is no transaction in progress
2026-09-25 07:31:24,150 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-25 07:31:24,151 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1eZXJS/build/release/bin/postgresql.jar:/tmp/dist-test-task1eZXJS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.66.20:35737/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-25 07:31:24,374 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-25 07:31:24,375 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1eZXJS/build/release/bin/postgresql.jar:/tmp/dist-test-task1eZXJS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.66.20:35737/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 07:31:24,613 [I] Table x_portal_user already exists in database 'ranger'
2026-09-25 07:31:24,613 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1eZXJS/build/release/bin/postgresql.jar:/tmp/dist-test-task1eZXJS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.66.20:35737/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 07:31:24,835 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-25 07:31:24,836 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1eZXJS/build/release/bin/postgresql.jar:/tmp/dist-test-task1eZXJS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.66.20:35737/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 07:31:25,083 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1eZXJS/build/release/bin/postgresql.jar:/tmp/dist-test-task1eZXJS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.66.20:35737/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 07:31:25,330 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1eZXJS/build/release/bin/postgresql.jar:/tmp/dist-test-task1eZXJS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.66.20:35737/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-57fn.c.gcp-upstream.internal';"
2026-09-25 07:31:25,550 [I] CORE_DB_SCHEMA import status has been updated
2026-09-25 07:31:25,550 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1eZXJS/build/release/bin/postgresql.jar:/tmp/dist-test-task1eZXJS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.66.20:35737/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-25 07:31:25,808 [I] DB_PATCHES have already been applied
I20260925 07:31:25.813493 9480 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/minicluster-data/ranger-admin:/tmp/dist-test-task1eZXJS/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task1eZXJS/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task1eZXJS/build/release/bin/postgresql.jar:/tmp/dist-test-task1eZXJS/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260925 07:31:25.813728 9480 mini_ranger.cc:193] Using host: 127.9.66.20
I20260925 07:31:25.816661 9480 mini_ranger.cc:227] Ranger admin URL: http://127.9.66.20:52213
Sep 25, 2026 7:31:26 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 7:31:26 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 7:31:26 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/minicluster-data/ranger-admin/ews/webapp
Sep 25, 2026 7:31:26 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 25, 2026 7:31:26 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 25, 2026 7:31:26 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/minicluster-data/ranger-admin/ews/webapp].
Sep 25, 2026 7:31:26 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260925 07:31:26.905797 9480 mini_ranger.cc:160] Time spent starting Ranger: real 8.157s user 0.003s sys 0.006s
Sep 25, 2026 7:31:46 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 25, 2026 7:31:47 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 25, 2026 7:31:47 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 25, 2026 7:31:47 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 25, 2026 7:31:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 25, 2026 7:31:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 25, 2026 7:31:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 25, 2026 7:31:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 25, 2026 7:31:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 25, 2026 7:31:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 25, 2026 7:31:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 25, 2026 7:31:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 25, 2026 7:31:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 25, 2026 7:31:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 25, 2026 7:31:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 25, 2026 7:31:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 25, 2026 7:31:47 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 25, 2026 7:31:47 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260925 07:31:48.542477 9480 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.9.66.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.9.66.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.9.66.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/krb5kdc/rangerkms_127.9.66.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.9.66.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/krb5kdc/rangerkms_127.9.66.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.9.66.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.9.66.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.66.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/krb5kdc/HTTP_127.9.66.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.9.66.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/krb5kdc/HTTP_127.9.66.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 25 07:31:48 dist-test-slave-57fn krb5kdc[20239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321508, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260925 07:31:48.629112 9480 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260925 07:31:48.637643 9480 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260925 07:31:49.468322 9480 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260925 07:31:49.468410 9480 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/minicluster-data/ranger-kms
I20260925 07:31:49.468428 9480 mini_ranger_kms.cc:227] Using postgres at 127.9.66.20:35737
2026-09-25 07:31:49,632 [I] DB FLAVOR :POSTGRES
2026-09-25 07:31:49,633 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 07:31:49,633 [I] Checking connection
2026-09-25 07:31:49,633 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1eZXJS/build/release/bin/postgresql.jar:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.66.20:35737/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-25 07:31:49,869 [I] connection success
2026-09-25 07:31:49,869 [I] --------- Verifying Ranger DB tables ---------
2026-09-25 07:31:49,869 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-25 07:31:49,869 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1eZXJS/build/release/bin/postgresql.jar:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.66.20:35737/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-25 07:31:50,085 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-25 07:31:50,085 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 07:31:50,085 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-25 07:31:50,085 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1eZXJS/build/release/bin/postgresql.jar:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.66.20:35737/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-25 07:31:50,343 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260925 07:31:50.780033 9480 mini_ranger_kms.cc:310] Created kms service
I20260925 07:31:50.981470 9480 mini_ranger_kms.cc:326] Created kudu user
I20260925 07:31:51.025758 9480 mini_ranger_kms.cc:343] Created rangerkms user
Sep 25 07:31:51 dist-test-slave-57fn krb5kdc[20239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321508, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.9.66.20@KRBTEST.COM
I20260925 07:31:51.424371 9480 mini_ranger_kms.cc:384] Added ranger policy
I20260925 07:31:51.424481 9480 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/minicluster-data/ranger-kms:/tmp/dist-test-task1eZXJS/build/release/bin/postgresql.jar:/tmp/dist-test-task1eZXJS/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task1eZXJS/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260925 07:31:51.424502 9480 mini_ranger_kms.cc:240] Using host: 127.9.66.20
I20260925 07:31:51.428231 9480 mini_ranger_kms.cc:276] Ranger KMS PID: 20632
I20260925 07:31:51.428337 9480 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.9.66.20:50155
Sep 25, 2026 7:31:52 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 7:31:52 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 7:31:52 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 25, 2026 7:31:52 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 25, 2026 7:31:53 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/minicluster-data/ranger-kms/ews/webapp].
Sep 25, 2026 7:31:53 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260925 07:31:53.727849 9480 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.099s user 0.002s sys 0.145s
I20260925 07:31:53.727952 9480 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260925 07:31:53.728024 9480 mini_ranger_kms.cc:401] 127.9.66.20:50155/kms/v1/keys
Sep 25 07:31:53 dist-test-slave-57fn krb5kdc[20239](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790321513, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.9.66.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 25 07:31:54 dist-test-slave-57fn krb5kdc[20239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321513, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.9.66.20@KRBTEST.COM for HTTP/127.9.66.20@KRBTEST.COM
Sep 25 07:31:54 dist-test-slave-57fn krb5kdc[20239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321513, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.9.66.20@KRBTEST.COM for HTTP/127.9.66.20@KRBTEST.COM
Sep 25 07:31:57 dist-test-slave-57fn krb5kdc[20239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321517, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.9.66.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.66.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.66.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab.
Entry for principal kudu/127.9.66.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab.
I20260925 07:31:57.109638 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.66.62:38097
--webserver_interface=127.9.66.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab
--principal=kudu/127.9.66.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:33059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.66.62:38097
--ranger_config_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.66.20:50155/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:31:57.202632 20683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:31:57.202814 20683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:31:57.202834 20683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:31:57.203908 20683 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:31:57.203943 20683 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:31:57.203954 20683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:31:57.203964 20683 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:31:57.203974 20683 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:31:57.205379 20683 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:33059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/minicluster-data/master-0/wal
--ranger_kms_url=127.9.66.20:50155/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.9.66.62:38097
--ranger_config_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab
--principal=kudu/127.9.66.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.66.62:38097
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.66.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20683
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:31:57.205595 20683 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:31:57.205794 20683 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:31:57.210165 20689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:31:57.210350 20683 server_base.cc:1034] running on GCE node
W20260925 07:31:57.210151 20688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:31:57.210170 20691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:31:57.210615 20683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:31:57.210891 20683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:31:57.212033 20683 hybrid_clock.cc:648] HybridClock initialized: now 1790321517212009 us; error 43 us; skew 500 ppm
Sep 25 07:31:57 dist-test-slave-57fn krb5kdc[20239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321517, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:31:57.218915 20683 init.cc:377] Logged in from keytab as kudu/127.9.66.62@KRBTEST.COM (short username kudu)
I20260925 07:31:57.219398 20683 webserver.cc:460] Webserver started at http://127.9.66.62:37947/ using document root <none> and password file <none>
I20260925 07:31:57.219619 20683 fs_manager.cc:359] Metadata directory not provided
I20260925 07:31:57.219676 20683 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:31:57.219825 20683 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 07:31:57 dist-test-slave-57fn krb5kdc[20239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321517, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.62@KRBTEST.COM for HTTP/127.9.66.20@KRBTEST.COM
I20260925 07:31:57.274984 20683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/minicluster-data/master-0/data/instance:
uuid: "c9e1263488934ad79d7faf635062ea01"
format_stamp: "Formatted at 2026-09-25 07:31:57 on dist-test-slave-57fn"
server_key: "3195f676e882c0640af101a8887535e2"
server_key_iv: "9b96d452dbf57bf52981b63c8bcc9ca1"
server_key_version: "kuduclusterkey@0"
I20260925 07:31:57.275430 20683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/minicluster-data/master-0/wal/instance:
uuid: "c9e1263488934ad79d7faf635062ea01"
format_stamp: "Formatted at 2026-09-25 07:31:57 on dist-test-slave-57fn"
server_key: "3195f676e882c0640af101a8887535e2"
server_key_iv: "9b96d452dbf57bf52981b63c8bcc9ca1"
server_key_version: "kuduclusterkey@0"
I20260925 07:31:57.277261 20683 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260925 07:31:57.290867 20698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:31:57.291110 20683 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 07:31:57.291236 20683 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/minicluster-data/master-0/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/minicluster-data/master-0/wal
uuid: "c9e1263488934ad79d7faf635062ea01"
format_stamp: "Formatted at 2026-09-25 07:31:57 on dist-test-slave-57fn"
server_key: "3195f676e882c0640af101a8887535e2"
server_key_iv: "9b96d452dbf57bf52981b63c8bcc9ca1"
server_key_version: "kuduclusterkey@0"
I20260925 07:31:57.291363 20683 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:31:57.335539 20683 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:31:57.337612 20683 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:31:57.337759 20683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:31:57.342761 20683 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.62:38097
I20260925 07:31:57.342790 20750 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.62:38097 every 8 connection(s)
I20260925 07:31:57.343189 20683 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
I20260925 07:31:57.343859 20751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:31:57.346092 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 20683
I20260925 07:31:57.346184 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/minicluster-data/master-0/wal/instance
I20260925 07:31:57.346144 20751 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9e1263488934ad79d7faf635062ea01: Bootstrap starting.
I20260925 07:31:57.346364 9480 external_mini_cluster.cc:1428] Setting key 1bbfdc5cc2a8ea4e20db2b82a25f1fc8
I20260925 07:31:57.347290 20751 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c9e1263488934ad79d7faf635062ea01: Neither blocks nor log segments found. Creating new log.
I20260925 07:31:57.347607 20751 log.cc:824] T 00000000000000000000000000000000 P c9e1263488934ad79d7faf635062ea01: Log is configured to *not* fsync() on all Append() calls
I20260925 07:31:57.349042 20751 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9e1263488934ad79d7faf635062ea01: No bootstrap required, opened a new log
I20260925 07:31:57.351086 20751 raft_consensus.cc:348] T 00000000000000000000000000000000 P c9e1263488934ad79d7faf635062ea01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9e1263488934ad79d7faf635062ea01" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 38097 } }
I20260925 07:31:57.351269 20751 raft_consensus.cc:374] T 00000000000000000000000000000000 P c9e1263488934ad79d7faf635062ea01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:31:57.351300 20751 raft_consensus.cc:729] T 00000000000000000000000000000000 P c9e1263488934ad79d7faf635062ea01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9e1263488934ad79d7faf635062ea01, State: Initialized, Role: FOLLOWER
I20260925 07:31:57.351415 20751 consensus_queue.cc:260] T 00000000000000000000000000000000 P c9e1263488934ad79d7faf635062ea01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9e1263488934ad79d7faf635062ea01" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 38097 } }
I20260925 07:31:57.351490 20751 raft_consensus.cc:388] T 00000000000000000000000000000000 P c9e1263488934ad79d7faf635062ea01 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 07:31:57.351519 20751 raft_consensus.cc:482] T 00000000000000000000000000000000 P c9e1263488934ad79d7faf635062ea01 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 07:31:57.351560 20751 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c9e1263488934ad79d7faf635062ea01 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:31:57.352393 20751 raft_consensus.cc:504] T 00000000000000000000000000000000 P c9e1263488934ad79d7faf635062ea01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9e1263488934ad79d7faf635062ea01" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 38097 } }
I20260925 07:31:57.352506 20751 leader_election.cc:302] T 00000000000000000000000000000000 P c9e1263488934ad79d7faf635062ea01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c9e1263488934ad79d7faf635062ea01; no voters:
I20260925 07:31:57.352684 20751 leader_election.cc:288] T 00000000000000000000000000000000 P c9e1263488934ad79d7faf635062ea01 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 07:31:57.352749 20756 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c9e1263488934ad79d7faf635062ea01 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:31:57.352898 20756 raft_consensus.cc:686] T 00000000000000000000000000000000 P c9e1263488934ad79d7faf635062ea01 [term 1 LEADER]: Becoming Leader. State: Replica: c9e1263488934ad79d7faf635062ea01, State: Running, Role: LEADER
I20260925 07:31:57.352941 20751 sys_catalog.cc:564] T 00000000000000000000000000000000 P c9e1263488934ad79d7faf635062ea01 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:31:57.352994 20756 consensus_queue.cc:237] T 00000000000000000000000000000000 P c9e1263488934ad79d7faf635062ea01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9e1263488934ad79d7faf635062ea01" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 38097 } }
I20260925 07:31:57.353320 20756 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9e1263488934ad79d7faf635062ea01 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c9e1263488934ad79d7faf635062ea01. Latest consensus state: current_term: 1 leader_uuid: "c9e1263488934ad79d7faf635062ea01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9e1263488934ad79d7faf635062ea01" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 38097 } } }
I20260925 07:31:57.353396 20756 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9e1263488934ad79d7faf635062ea01 [sys.catalog]: This master's current role is: LEADER
I20260925 07:31:57.353344 20757 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9e1263488934ad79d7faf635062ea01 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c9e1263488934ad79d7faf635062ea01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9e1263488934ad79d7faf635062ea01" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 38097 } } }
I20260925 07:31:57.353582 20757 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9e1263488934ad79d7faf635062ea01 [sys.catalog]: This master's current role is: LEADER
I20260925 07:31:57.354278 20761 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 25 07:31:57 dist-test-slave-57fn krb5kdc[20239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321517, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.66.62@KRBTEST.COM
I20260925 07:31:57.354573 20751 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 07:31:57.354755 20761 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 07:31:57.361238 20754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:31:57.347015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43950 (local address 127.9.66.62:38097)
0925 07:31:57.347184 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 07:31:57.347194 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 07:31:57.347226 (+ 32us) server_negotiation.cc:407] Connection header received
0925 07:31:57.347842 (+ 616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:31:57.347851 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:31:57.347914 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:31:57.348062 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:31:57.348737 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:31:57.350199 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:31:57.351164 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:31:57.351820 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:31:57.351940 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:31:57.354943 (+ 3003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:31:57.354947 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:31:57.354947 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:31:57.354958 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:31:57.359517 (+ 4559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:31:57.359944 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:31:57.359947 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:31:57.359947 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:31:57.360035 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:31:57.360237 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:31:57.360238 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:31:57.360239 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:31:57.360476 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:31:57.360644 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0925 07:31:57.360970 (+ 326us) server_negotiation.cc:299] Negotiation successful
0925 07:31:57.361098 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":83,"threads_started":1}
I20260925 07:31:57.361919 20761 catalog_manager.cc:1347] Generated new cluster ID: 7ea36e91b02b4f019242e0e3c35e202c
I20260925 07:31:57.361972 20761 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 07:31:57.381639 20761 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 07:31:57.382424 20761 catalog_manager.cc:1504] Loading token signing keys...
I20260925 07:31:57.385404 20761 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c9e1263488934ad79d7faf635062ea01: Generated new TSK 0
I20260925 07:31:57.385782 20761 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.66.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.66.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.66.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.1.keytab.
Entry for principal kudu/127.9.66.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.1.keytab.
I20260925 07:31:58.076687 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.66.20:50155/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.66.1:0
--local_ip_for_outbound_sockets=127.9.66.1
--webserver_interface=127.9.66.1
--webserver_port=0
--tserver_master_addrs=127.9.66.62:38097
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.1.keytab
--principal=kudu/127.9.66.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:33059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:31:58.181876 20796 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:31:58.182176 20796 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:31:58.182245 20796 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:31:58.183964 20796 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:31:58.184079 20796 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:31:58.184139 20796 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.66.1
I20260925 07:31:58.185839 20796 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:33059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.9.66.20:50155/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.1.keytab
--principal=kudu/127.9.66.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.66.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.66.1
--webserver_port=0
--tserver_master_addrs=127.9.66.62:38097
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20796
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.66.1
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:31:58.186293 20796 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:31:58.186546 20796 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:31:58.193123 20804 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:31:58.193238 20796 server_base.cc:1034] running on GCE node
W20260925 07:31:58.193339 20806 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:31:58.193449 20803 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:31:58.193704 20796 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:31:58.193958 20796 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:31:58.195124 20796 hybrid_clock.cc:648] HybridClock initialized: now 1790321518195105 us; error 33 us; skew 500 ppm
Sep 25 07:31:58 dist-test-slave-57fn krb5kdc[20239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321518, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:31:58.198221 20796 init.cc:377] Logged in from keytab as kudu/127.9.66.1@KRBTEST.COM (short username kudu)
I20260925 07:31:58.198725 20796 webserver.cc:460] Webserver started at http://127.9.66.1:34173/ using document root <none> and password file <none>
I20260925 07:31:58.198949 20796 fs_manager.cc:359] Metadata directory not provided
I20260925 07:31:58.199024 20796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:31:58.199157 20796 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 07:31:58 dist-test-slave-57fn krb5kdc[20239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321518, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.1@KRBTEST.COM for HTTP/127.9.66.20@KRBTEST.COM
I20260925 07:31:58.212952 20796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/minicluster-data/ts-0/data/instance:
uuid: "27d46d7f2cbe4b80bd823e1824c4a324"
format_stamp: "Formatted at 2026-09-25 07:31:58 on dist-test-slave-57fn"
server_key: "792c020e5e1f8e11458171f2ff0c2ee8"
server_key_iv: "53ae5298c88022b29deb0758a2718d9d"
server_key_version: "kuduclusterkey@0"
I20260925 07:31:58.213483 20796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/minicluster-data/ts-0/wal/instance:
uuid: "27d46d7f2cbe4b80bd823e1824c4a324"
format_stamp: "Formatted at 2026-09-25 07:31:58 on dist-test-slave-57fn"
server_key: "792c020e5e1f8e11458171f2ff0c2ee8"
server_key_iv: "53ae5298c88022b29deb0758a2718d9d"
server_key_version: "kuduclusterkey@0"
I20260925 07:31:58.215344 20796 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260925 07:31:58.235895 20813 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:31:58.236193 20796 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.001s sys 0.000s
I20260925 07:31:58.236302 20796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/minicluster-data/ts-0/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/minicluster-data/ts-0/wal
uuid: "27d46d7f2cbe4b80bd823e1824c4a324"
format_stamp: "Formatted at 2026-09-25 07:31:58 on dist-test-slave-57fn"
server_key: "792c020e5e1f8e11458171f2ff0c2ee8"
server_key_iv: "53ae5298c88022b29deb0758a2718d9d"
server_key_version: "kuduclusterkey@0"
I20260925 07:31:58.236433 20796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:31:58.253324 20796 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:31:58.255669 20796 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:31:58.255839 20796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:31:58.256088 20796 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:31:58.256409 20796 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:31:58.256448 20796 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:31:58.256474 20796 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:31:58.256525 20796 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:31:58.263708 20796 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.1:35135
I20260925 07:31:58.263826 20926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.1:35135 every 8 connection(s)
I20260925 07:31:58.264189 20796 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/minicluster-data/ts-0/data/info.pb
Sep 25 07:31:58 dist-test-slave-57fn krb5kdc[20239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321518, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.1@KRBTEST.COM for kudu/127.9.66.62@KRBTEST.COM
I20260925 07:31:58.273540 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 20796
I20260925 07:31:58.273649 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/minicluster-data/ts-0/wal/instance
I20260925 07:31:58.273841 9480 external_mini_cluster.cc:1428] Setting key 530628247435a43b6fab5bd8d52604c2
I20260925 07:31:58.275357 20929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:31:58.264879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:38097 (local address 127.9.66.1:59383)
0925 07:31:58.265082 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0925 07:31:58.265148 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0925 07:31:58.265846 (+ 698us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:31:58.266224 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:31:58.266228 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:31:58.266439 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:31:58.266700 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:31:58.266706 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:31:58.267740 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:31:58.267741 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:31:58.268983 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:31:58.268986 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:31:58.269541 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:31:58.269542 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:31:58.269673 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:31:58.270252 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:31:58.270261 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:31:58.271820 (+ 1559us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:31:58.273825 (+ 2005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:31:58.273832 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:31:58.273834 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:31:58.274118 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:31:58.274358 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:31:58.274359 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:31:58.274359 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:31:58.274470 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:31:58.274745 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:31:58.274750 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:31:58.274908 (+ 158us) client_negotiation.cc:769] Sending connection context
0925 07:31:58.275009 (+ 101us) client_negotiation.cc:240] Negotiation successful
0925 07:31:58.275141 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":94,"threads_started":1}
I20260925 07:31:58.275943 20930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:31:58.264900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.1:59383 (local address 127.9.66.62:38097)
0925 07:31:58.265079 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 07:31:58.265082 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:31:58.265979 (+ 897us) server_negotiation.cc:407] Connection header received
0925 07:31:58.266010 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:31:58.266011 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:31:58.266026 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:31:58.266107 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:31:58.266806 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:31:58.267637 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:31:58.269082 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:31:58.269398 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:31:58.269452 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:31:58.271922 (+ 2470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:31:58.271924 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:31:58.271924 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:31:58.271931 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:31:58.273735 (+ 1804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:31:58.274219 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:31:58.274220 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:31:58.274221 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:31:58.274295 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:31:58.274543 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:31:58.274544 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:31:58.274544 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:31:58.274664 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:31:58.274721 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 07:31:58.275768 (+ 1047us) server_negotiation.cc:299] Negotiation successful
0925 07:31:58.275849 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":91,"threads_started":1}
I20260925 07:31:58.276690 20927 heartbeater.cc:347] Connected to a master server at 127.9.66.62:38097
I20260925 07:31:58.276777 20927 heartbeater.cc:464] Registering TS with master...
I20260925 07:31:58.276950 20927 heartbeater.cc:511] Master 127.9.66.62:38097 requested a full tablet report, sending...
I20260925 07:31:58.277523 20715 ts_manager.cc:194] Registered new tserver with Master: 27d46d7f2cbe4b80bd823e1824c4a324 (127.9.66.1:35135)
I20260925 07:31:58.278407 20715 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.66.1@KRBTEST.COM'} at 127.9.66.1:59383
WARNING: no policy specified for kudu/127.9.66.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.66.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.66.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.2.keytab.
Entry for principal kudu/127.9.66.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.2.keytab.
I20260925 07:31:58.308741 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.66.20:50155/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.66.2:0
--local_ip_for_outbound_sockets=127.9.66.2
--webserver_interface=127.9.66.2
--webserver_port=0
--tserver_master_addrs=127.9.66.62:38097
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.2.keytab
--principal=kudu/127.9.66.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:33059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 07:31:58 dist-test-slave-57fn krb5kdc[20239](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790321518, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260925 07:31:58.402945 20935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:31:58.403272 20935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:31:58.403335 20935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:31:58.405042 20935 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:31:58.405098 20935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:31:58.405133 20935 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.66.2
I20260925 07:31:58.407274 20935 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:33059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.9.66.20:50155/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.2.keytab
--principal=kudu/127.9.66.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.66.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.66.2
--webserver_port=0
--tserver_master_addrs=127.9.66.62:38097
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20935
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.66.2
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:31:58.407642 20935 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:31:58.408066 20935 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:31:58.414309 20941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:31:58.414578 20943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:31:58.414747 20940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:31:58.414800 20935 server_base.cc:1034] running on GCE node
I20260925 07:31:58.415061 20935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:31:58.415333 20935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:31:58.416486 20935 hybrid_clock.cc:648] HybridClock initialized: now 1790321518416463 us; error 39 us; skew 500 ppm
Sep 25 07:31:58 dist-test-slave-57fn krb5kdc[20239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321518, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:31:58.419003 20935 init.cc:377] Logged in from keytab as kudu/127.9.66.2@KRBTEST.COM (short username kudu)
I20260925 07:31:58.419487 20935 webserver.cc:460] Webserver started at http://127.9.66.2:34769/ using document root <none> and password file <none>
I20260925 07:31:58.419715 20935 fs_manager.cc:359] Metadata directory not provided
I20260925 07:31:58.419759 20935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:31:58.419894 20935 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 07:31:58 dist-test-slave-57fn krb5kdc[20239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321518, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.2@KRBTEST.COM for HTTP/127.9.66.20@KRBTEST.COM
I20260925 07:31:58.432010 20935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/minicluster-data/ts-1/data/instance:
uuid: "0c1b4daf69fe439a9b9fd25e32921420"
format_stamp: "Formatted at 2026-09-25 07:31:58 on dist-test-slave-57fn"
server_key: "6d17f5576fcd7216a65cd32a1c094a90"
server_key_iv: "74389d074bfce584e361bc90cf7c0f23"
server_key_version: "kuduclusterkey@0"
I20260925 07:31:58.432411 20935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/minicluster-data/ts-1/wal/instance:
uuid: "0c1b4daf69fe439a9b9fd25e32921420"
format_stamp: "Formatted at 2026-09-25 07:31:58 on dist-test-slave-57fn"
server_key: "6d17f5576fcd7216a65cd32a1c094a90"
server_key_iv: "74389d074bfce584e361bc90cf7c0f23"
server_key_version: "kuduclusterkey@0"
I20260925 07:31:58.434505 20935 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260925 07:31:58.452963 20950 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:31:58.453233 20935 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.000s sys 0.000s
I20260925 07:31:58.453330 20935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/minicluster-data/ts-1/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/minicluster-data/ts-1/wal
uuid: "0c1b4daf69fe439a9b9fd25e32921420"
format_stamp: "Formatted at 2026-09-25 07:31:58 on dist-test-slave-57fn"
server_key: "6d17f5576fcd7216a65cd32a1c094a90"
server_key_iv: "74389d074bfce584e361bc90cf7c0f23"
server_key_version: "kuduclusterkey@0"
I20260925 07:31:58.453441 20935 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:31:58.466912 20935 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:31:58.469431 20935 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:31:58.469663 20935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:31:58.469930 20935 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:31:58.470337 20935 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:31:58.470465 20935 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:31:58.470537 20935 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:31:58.470595 20935 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:31:58.476969 20935 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.2:37557
I20260925 07:31:58.477411 20935 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/minicluster-data/ts-1/data/info.pb
I20260925 07:31:58.480377 21064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.2:37557 every 8 connection(s)
I20260925 07:31:58.486861 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 20935
I20260925 07:31:58.486976 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/minicluster-data/ts-1/wal/instance
I20260925 07:31:58.487207 9480 external_mini_cluster.cc:1428] Setting key 473ddf7d45e7583c8c76f900362360ba
Sep 25 07:31:58 dist-test-slave-57fn krb5kdc[20239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321518, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.2@KRBTEST.COM for kudu/127.9.66.62@KRBTEST.COM
I20260925 07:31:58.492137 20930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:31:58.481184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.2:40957 (local address 127.9.66.62:38097)
0925 07:31:58.481236 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 07:31:58.481238 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:31:58.481490 (+ 252us) server_negotiation.cc:407] Connection header received
0925 07:31:58.482267 (+ 777us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:31:58.482268 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:31:58.482281 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:31:58.482312 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:31:58.483154 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:31:58.483965 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:31:58.485468 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:31:58.485823 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:31:58.485867 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:31:58.488721 (+ 2854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:31:58.488722 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:31:58.488723 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:31:58.488730 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:31:58.490508 (+ 1778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:31:58.491007 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:31:58.491008 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:31:58.491009 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:31:58.491069 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:31:58.491353 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:31:58.491354 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:31:58.491354 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:31:58.491505 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:31:58.491553 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 07:31:58.491950 (+ 397us) server_negotiation.cc:299] Negotiation successful
0925 07:31:58.492042 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 07:31:58.492897 21067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:31:58.481163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:38097 (local address 127.9.66.2:40957)
0925 07:31:58.481343 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0925 07:31:58.481399 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0925 07:31:58.482135 (+ 736us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:31:58.482520 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:31:58.482524 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:31:58.482761 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:31:58.483031 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:31:58.483038 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:31:58.484085 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:31:58.484086 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:31:58.485350 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:31:58.485353 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:31:58.485964 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:31:58.485965 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:31:58.486101 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:31:58.486702 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:31:58.486712 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:31:58.488607 (+ 1895us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:31:58.490607 (+ 2000us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:31:58.490614 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:31:58.490617 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:31:58.490912 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:31:58.491142 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:31:58.491143 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:31:58.491143 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:31:58.491273 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:31:58.491604 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:31:58.491610 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:31:58.491780 (+ 170us) client_negotiation.cc:769] Sending connection context
0925 07:31:58.492493 (+ 713us) client_negotiation.cc:240] Negotiation successful
0925 07:31:58.492663 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":98,"threads_started":1}
I20260925 07:31:58.493661 21065 heartbeater.cc:347] Connected to a master server at 127.9.66.62:38097
I20260925 07:31:58.493768 21065 heartbeater.cc:464] Registering TS with master...
I20260925 07:31:58.493947 21065 heartbeater.cc:511] Master 127.9.66.62:38097 requested a full tablet report, sending...
I20260925 07:31:58.494377 20715 ts_manager.cc:194] Registered new tserver with Master: 0c1b4daf69fe439a9b9fd25e32921420 (127.9.66.2:37557)
I20260925 07:31:58.494848 20715 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.66.2@KRBTEST.COM'} at 127.9.66.2:40957
WARNING: no policy specified for kudu/127.9.66.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.66.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.66.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.3.keytab.
Entry for principal kudu/127.9.66.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.3.keytab.
I20260925 07:31:58.534106 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.66.20:50155/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.66.3:0
--local_ip_for_outbound_sockets=127.9.66.3
--webserver_interface=127.9.66.3
--webserver_port=0
--tserver_master_addrs=127.9.66.62:38097
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.3.keytab
--principal=kudu/127.9.66.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:33059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:31:58.662798 21070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:31:58.663029 21070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:31:58.663069 21070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:31:58.664359 21070 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:31:58.664428 21070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:31:58.664474 21070 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.66.3
I20260925 07:31:58.666009 21070 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:33059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.9.66.20:50155/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.3.keytab
--principal=kudu/127.9.66.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.66.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.66.3
--webserver_port=0
--tserver_master_addrs=127.9.66.62:38097
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21070
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.66.3
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:31:58.666290 21070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:31:58.666530 21070 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:31:58.671389 21075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:31:58.671561 21076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:31:58.671434 21078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:31:58.671881 21070 server_base.cc:1034] running on GCE node
I20260925 07:31:58.672087 21070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:31:58.672329 21070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:31:58.675666 21070 hybrid_clock.cc:648] HybridClock initialized: now 1790321518675652 us; error 26 us; skew 500 ppm
Sep 25 07:31:58 dist-test-slave-57fn krb5kdc[20239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321518, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:31:58.678179 21070 init.cc:377] Logged in from keytab as kudu/127.9.66.3@KRBTEST.COM (short username kudu)
I20260925 07:31:58.678663 21070 webserver.cc:460] Webserver started at http://127.9.66.3:37135/ using document root <none> and password file <none>
I20260925 07:31:58.678891 21070 fs_manager.cc:359] Metadata directory not provided
I20260925 07:31:58.678956 21070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:31:58.679095 21070 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 07:31:58 dist-test-slave-57fn krb5kdc[20239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321518, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.3@KRBTEST.COM for HTTP/127.9.66.20@KRBTEST.COM
I20260925 07:31:58.695749 21070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/minicluster-data/ts-2/data/instance:
uuid: "4bb868e1eb6e48ec8edd9c0fe78f1f5f"
format_stamp: "Formatted at 2026-09-25 07:31:58 on dist-test-slave-57fn"
server_key: "da91148ee274aef938f5ed3629267542"
server_key_iv: "deb1bbe1404443e9b16f3a2bdf079f73"
server_key_version: "kuduclusterkey@0"
I20260925 07:31:58.696297 21070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/minicluster-data/ts-2/wal/instance:
uuid: "4bb868e1eb6e48ec8edd9c0fe78f1f5f"
format_stamp: "Formatted at 2026-09-25 07:31:58 on dist-test-slave-57fn"
server_key: "da91148ee274aef938f5ed3629267542"
server_key_iv: "deb1bbe1404443e9b16f3a2bdf079f73"
server_key_version: "kuduclusterkey@0"
I20260925 07:31:58.699422 21070 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.001s
I20260925 07:31:58.722787 21085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:31:58.723045 21070 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 07:31:58.723142 21070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/minicluster-data/ts-2/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/minicluster-data/ts-2/wal
uuid: "4bb868e1eb6e48ec8edd9c0fe78f1f5f"
format_stamp: "Formatted at 2026-09-25 07:31:58 on dist-test-slave-57fn"
server_key: "da91148ee274aef938f5ed3629267542"
server_key_iv: "deb1bbe1404443e9b16f3a2bdf079f73"
server_key_version: "kuduclusterkey@0"
I20260925 07:31:58.723229 21070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:31:58.736348 21070 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:31:58.738715 21070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:31:58.738907 21070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:31:58.739169 21070 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:31:58.739588 21070 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:31:58.739677 21070 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:31:58.739748 21070 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:31:58.739805 21070 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:31:58.747133 21070 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.3:38533
I20260925 07:31:58.747602 21070 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/minicluster-data/ts-2/data/info.pb
I20260925 07:31:58.749379 21198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.3:38533 every 8 connection(s)
I20260925 07:31:58.754511 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 21070
I20260925 07:31:58.754623 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/minicluster-data/ts-2/wal/instance
I20260925 07:31:58.754827 9480 external_mini_cluster.cc:1428] Setting key f0bb3ea4c85e84d312dfc71c030c5f68
Sep 25 07:31:58 dist-test-slave-57fn krb5kdc[20239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321518, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.3@KRBTEST.COM for kudu/127.9.66.62@KRBTEST.COM
I20260925 07:31:58.764524 20930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:31:58.749983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.3:53835 (local address 127.9.66.62:38097)
0925 07:31:58.750030 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 07:31:58.750032 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:31:58.753821 (+ 3789us) server_negotiation.cc:407] Connection header received
0925 07:31:58.754724 (+ 903us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:31:58.754726 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:31:58.754741 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:31:58.754775 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:31:58.755646 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:31:58.756444 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:31:58.758114 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:31:58.758399 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:31:58.758439 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:31:58.761051 (+ 2612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:31:58.761053 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:31:58.761053 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:31:58.761060 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:31:58.762831 (+ 1771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:31:58.763357 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:31:58.763358 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:31:58.763358 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:31:58.763422 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:31:58.763731 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:31:58.763732 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:31:58.763732 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:31:58.763874 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:31:58.763919 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0925 07:31:58.764336 (+ 417us) server_negotiation.cc:299] Negotiation successful
0925 07:31:58.764422 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 07:31:58.765342 21201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:31:58.749995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:38097 (local address 127.9.66.3:53835)
0925 07:31:58.753648 (+ 3653us) negotiation.cc:107] Waiting for socket to connect
0925 07:31:58.753707 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 07:31:58.754570 (+ 863us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:31:58.754964 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:31:58.754969 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:31:58.755228 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:31:58.755529 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:31:58.755535 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:31:58.756591 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:31:58.756591 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:31:58.758013 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:31:58.758016 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:31:58.758526 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:31:58.758527 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:31:58.758659 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:31:58.759277 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:31:58.759288 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:31:58.760927 (+ 1639us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:31:58.762933 (+ 2006us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:31:58.762940 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:31:58.762943 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:31:58.763259 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:31:58.763501 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:31:58.763501 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:31:58.763502 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:31:58.763640 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:31:58.763968 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:31:58.763988 (+ 20us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:31:58.764173 (+ 185us) client_negotiation.cc:769] Sending connection context
0925 07:31:58.764915 (+ 742us) client_negotiation.cc:240] Negotiation successful
0925 07:31:58.765102 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3598,"thread_start_us":102,"threads_started":1}
I20260925 07:31:58.766153 21199 heartbeater.cc:347] Connected to a master server at 127.9.66.62:38097
I20260925 07:31:58.766258 21199 heartbeater.cc:464] Registering TS with master...
I20260925 07:31:58.766456 21199 heartbeater.cc:511] Master 127.9.66.62:38097 requested a full tablet report, sending...
I20260925 07:31:58.767009 20715 ts_manager.cc:194] Registered new tserver with Master: 4bb868e1eb6e48ec8edd9c0fe78f1f5f (127.9.66.3:38533)
I20260925 07:31:58.767527 20715 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.66.3@KRBTEST.COM'} at 127.9.66.3:53835
I20260925 07:31:58.778542 9480 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 07:31:58.788162 20930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:31:58.780201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43964 (local address 127.9.66.62:38097)
0925 07:31:58.780244 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 07:31:58.780246 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:31:58.780537 (+ 291us) server_negotiation.cc:407] Connection header received
0925 07:31:58.780629 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:31:58.780630 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:31:58.780640 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:31:58.780669 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:31:58.781425 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:31:58.782224 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:31:58.783343 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:31:58.783617 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:31:58.783663 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:31:58.785263 (+ 1600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:31:58.785264 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:31:58.785264 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:31:58.785271 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:31:58.786943 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:31:58.787383 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:31:58.787384 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:31:58.787385 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:31:58.787438 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:31:58.787638 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:31:58.787639 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:31:58.787639 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:31:58.787752 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:31:58.787818 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 07:31:58.788005 (+ 187us) server_negotiation.cc:299] Negotiation successful
0925 07:31:58.788076 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 07:31:58.790081 20715 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43964:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 07:31:58.792331 20715 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 07:31:58.805227 21214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:31:58.796400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.2:37557 (local address 127.0.0.1:46772)
0925 07:31:58.797905 (+ 1505us) negotiation.cc:107] Waiting for socket to connect
0925 07:31:58.797913 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 07:31:58.797947 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:31:58.798176 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:31:58.798177 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:31:58.798178 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:31:58.798229 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:31:58.798230 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:31:58.799777 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:31:58.799778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:31:58.803955 (+ 4177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:31:58.803957 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:31:58.804946 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:31:58.804947 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:31:58.805030 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:31:58.805042 (+ 12us) client_negotiation.cc:769] Sending connection context
0925 07:31:58.805101 (+ 59us) client_negotiation.cc:240] Negotiation successful
0925 07:31:58.805148 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1493,"thread_start_us":81,"threads_started":1}
I20260925 07:31:58.806396 21216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:31:58.795726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.3:38533 (local address 127.0.0.1:54380)
0925 07:31:58.797061 (+ 1335us) negotiation.cc:107] Waiting for socket to connect
0925 07:31:58.797099 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 07:31:58.797155 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:31:58.797746 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:31:58.797749 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:31:58.797751 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:31:58.797821 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:31:58.797823 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:31:58.802210 (+ 4387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:31:58.802211 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:31:58.803242 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:31:58.803245 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:31:58.806153 (+ 2908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:31:58.806154 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:31:58.806224 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:31:58.806227 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 07:31:58.806274 (+ 47us) client_negotiation.cc:240] Negotiation successful
0925 07:31:58.806324 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1307,"thread_start_us":68,"threads_started":1}
I20260925 07:31:58.806970 21217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:31:58.796641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46772 (local address 127.9.66.2:37557)
0925 07:31:58.797292 (+ 651us) server_negotiation.cc:206] Beginning negotiation
0925 07:31:58.797298 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 07:31:58.797997 (+ 699us) server_negotiation.cc:407] Connection header received
0925 07:31:58.798035 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:31:58.798037 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:31:58.798051 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:31:58.798110 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:31:58.798295 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:31:58.799683 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:31:58.804054 (+ 4371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:31:58.804775 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:31:58.804844 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:31:58.804914 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 07:31:58.806852 (+ 1938us) server_negotiation.cc:299] Negotiation successful
0925 07:31:58.806893 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":614,"thread_start_us":69,"threads_started":1}
I20260925 07:31:58.808044 20999 tablet_service.cc:1467] Processing CreateTablet for tablet 16707624138c46f69436701eec799546 (DEFAULT_TABLE table=test-table [id=b0ed3ee32ace4d6ba7ce605e430613e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:31:58.808501 20999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16707624138c46f69436701eec799546. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:31:58.809317 21212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:31:58.796195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.1:35135 (local address 127.0.0.1:39138)
0925 07:31:58.797349 (+ 1154us) negotiation.cc:107] Waiting for socket to connect
0925 07:31:58.797358 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 07:31:58.797392 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:31:58.805803 (+ 8411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:31:58.805804 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:31:58.805805 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:31:58.805853 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:31:58.805854 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:31:58.807087 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:31:58.807088 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:31:58.808169 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:31:58.808172 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:31:58.809124 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:31:58.809125 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:31:58.809177 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:31:58.809181 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 07:31:58.809223 (+ 42us) client_negotiation.cc:240] Negotiation successful
0925 07:31:58.809257 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1141,"thread_start_us":66,"threads_started":1}
I20260925 07:31:58.809595 21215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:31:58.796283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39138 (local address 127.9.66.1:35135)
0925 07:31:58.805591 (+ 9308us) server_negotiation.cc:206] Beginning negotiation
0925 07:31:58.805596 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 07:31:58.805606 (+ 10us) server_negotiation.cc:407] Connection header received
0925 07:31:58.805640 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:31:58.805642 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:31:58.805657 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:31:58.805735 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:31:58.805946 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:31:58.806977 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:31:58.808272 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:31:58.808980 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:31:58.809046 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:31:58.809093 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 07:31:58.809467 (+ 374us) server_negotiation.cc:299] Negotiation successful
0925 07:31:58.809504 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9270,"thread_start_us":52,"threads_started":1}
I20260925 07:31:58.810141 20861 tablet_service.cc:1467] Processing CreateTablet for tablet 16707624138c46f69436701eec799546 (DEFAULT_TABLE table=test-table [id=b0ed3ee32ace4d6ba7ce605e430613e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:31:58.810452 20861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16707624138c46f69436701eec799546. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:31:58.812929 21213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:31:58.795973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54380 (local address 127.9.66.3:38533)
0925 07:31:58.797499 (+ 1526us) server_negotiation.cc:206] Beginning negotiation
0925 07:31:58.797521 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0925 07:31:58.797531 (+ 10us) server_negotiation.cc:407] Connection header received
0925 07:31:58.797567 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:31:58.797570 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:31:58.797584 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:31:58.797650 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:31:58.801190 (+ 3540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:31:58.802124 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:31:58.803368 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:31:58.805998 (+ 2630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:31:58.806054 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:31:58.806122 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 07:31:58.812795 (+ 6673us) server_negotiation.cc:299] Negotiation successful
0925 07:31:58.812842 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1491,"thread_start_us":81,"threads_started":1}
I20260925 07:31:58.813719 21133 tablet_service.cc:1467] Processing CreateTablet for tablet 16707624138c46f69436701eec799546 (DEFAULT_TABLE table=test-table [id=b0ed3ee32ace4d6ba7ce605e430613e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:31:58.814013 21133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16707624138c46f69436701eec799546. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:31:58.817297 21218 tablet_bootstrap.cc:492] T 16707624138c46f69436701eec799546 P 0c1b4daf69fe439a9b9fd25e32921420: Bootstrap starting.
I20260925 07:31:58.817618 21220 tablet_bootstrap.cc:492] T 16707624138c46f69436701eec799546 P 4bb868e1eb6e48ec8edd9c0fe78f1f5f: Bootstrap starting.
I20260925 07:31:58.818424 21218 tablet_bootstrap.cc:654] T 16707624138c46f69436701eec799546 P 0c1b4daf69fe439a9b9fd25e32921420: Neither blocks nor log segments found. Creating new log.
I20260925 07:31:58.819324 21220 tablet_bootstrap.cc:654] T 16707624138c46f69436701eec799546 P 4bb868e1eb6e48ec8edd9c0fe78f1f5f: Neither blocks nor log segments found. Creating new log.
I20260925 07:31:58.819640 21218 log.cc:824] T 16707624138c46f69436701eec799546 P 0c1b4daf69fe439a9b9fd25e32921420: Log is configured to *not* fsync() on all Append() calls
I20260925 07:31:58.819952 21220 log.cc:824] T 16707624138c46f69436701eec799546 P 4bb868e1eb6e48ec8edd9c0fe78f1f5f: Log is configured to *not* fsync() on all Append() calls
I20260925 07:31:58.820063 21219 tablet_bootstrap.cc:492] T 16707624138c46f69436701eec799546 P 27d46d7f2cbe4b80bd823e1824c4a324: Bootstrap starting.
I20260925 07:31:58.820660 21219 tablet_bootstrap.cc:654] T 16707624138c46f69436701eec799546 P 27d46d7f2cbe4b80bd823e1824c4a324: Neither blocks nor log segments found. Creating new log.
I20260925 07:31:58.820950 21219 log.cc:824] T 16707624138c46f69436701eec799546 P 27d46d7f2cbe4b80bd823e1824c4a324: Log is configured to *not* fsync() on all Append() calls
I20260925 07:31:58.821099 21218 tablet_bootstrap.cc:492] T 16707624138c46f69436701eec799546 P 0c1b4daf69fe439a9b9fd25e32921420: No bootstrap required, opened a new log
I20260925 07:31:58.821259 21218 ts_tablet_manager.cc:1397] T 16707624138c46f69436701eec799546 P 0c1b4daf69fe439a9b9fd25e32921420: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260925 07:31:58.823333 21218 raft_consensus.cc:348] T 16707624138c46f69436701eec799546 P 0c1b4daf69fe439a9b9fd25e32921420 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c1b4daf69fe439a9b9fd25e32921420" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 37557 } } peers { permanent_uuid: "27d46d7f2cbe4b80bd823e1824c4a324" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 35135 } } peers { permanent_uuid: "4bb868e1eb6e48ec8edd9c0fe78f1f5f" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 38533 } }
I20260925 07:31:58.823572 21218 raft_consensus.cc:374] T 16707624138c46f69436701eec799546 P 0c1b4daf69fe439a9b9fd25e32921420 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:31:58.823647 21218 raft_consensus.cc:729] T 16707624138c46f69436701eec799546 P 0c1b4daf69fe439a9b9fd25e32921420 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c1b4daf69fe439a9b9fd25e32921420, State: Initialized, Role: FOLLOWER
I20260925 07:31:58.823848 21218 consensus_queue.cc:260] T 16707624138c46f69436701eec799546 P 0c1b4daf69fe439a9b9fd25e32921420 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c1b4daf69fe439a9b9fd25e32921420" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 37557 } } peers { permanent_uuid: "27d46d7f2cbe4b80bd823e1824c4a324" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 35135 } } peers { permanent_uuid: "4bb868e1eb6e48ec8edd9c0fe78f1f5f" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 38533 } }
I20260925 07:31:58.824127 21218 ts_tablet_manager.cc:1428] T 16707624138c46f69436701eec799546 P 0c1b4daf69fe439a9b9fd25e32921420: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260925 07:31:58.824266 21065 heartbeater.cc:503] Master 127.9.66.62:38097 was elected leader, sending a full tablet report...
I20260925 07:31:58.825412 21220 tablet_bootstrap.cc:492] T 16707624138c46f69436701eec799546 P 4bb868e1eb6e48ec8edd9c0fe78f1f5f: No bootstrap required, opened a new log
I20260925 07:31:58.825465 21220 ts_tablet_manager.cc:1397] T 16707624138c46f69436701eec799546 P 4bb868e1eb6e48ec8edd9c0fe78f1f5f: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20260925 07:31:58.827268 21220 raft_consensus.cc:348] T 16707624138c46f69436701eec799546 P 4bb868e1eb6e48ec8edd9c0fe78f1f5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c1b4daf69fe439a9b9fd25e32921420" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 37557 } } peers { permanent_uuid: "27d46d7f2cbe4b80bd823e1824c4a324" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 35135 } } peers { permanent_uuid: "4bb868e1eb6e48ec8edd9c0fe78f1f5f" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 38533 } }
I20260925 07:31:58.827461 21220 raft_consensus.cc:374] T 16707624138c46f69436701eec799546 P 4bb868e1eb6e48ec8edd9c0fe78f1f5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:31:58.827529 21220 raft_consensus.cc:729] T 16707624138c46f69436701eec799546 P 4bb868e1eb6e48ec8edd9c0fe78f1f5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bb868e1eb6e48ec8edd9c0fe78f1f5f, State: Initialized, Role: FOLLOWER
I20260925 07:31:58.827687 21220 consensus_queue.cc:260] T 16707624138c46f69436701eec799546 P 4bb868e1eb6e48ec8edd9c0fe78f1f5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c1b4daf69fe439a9b9fd25e32921420" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 37557 } } peers { permanent_uuid: "27d46d7f2cbe4b80bd823e1824c4a324" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 35135 } } peers { permanent_uuid: "4bb868e1eb6e48ec8edd9c0fe78f1f5f" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 38533 } }
I20260925 07:31:58.827946 21220 ts_tablet_manager.cc:1428] T 16707624138c46f69436701eec799546 P 4bb868e1eb6e48ec8edd9c0fe78f1f5f: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260925 07:31:58.828078 21199 heartbeater.cc:503] Master 127.9.66.62:38097 was elected leader, sending a full tablet report...
I20260925 07:31:58.829941 21219 tablet_bootstrap.cc:492] T 16707624138c46f69436701eec799546 P 27d46d7f2cbe4b80bd823e1824c4a324: No bootstrap required, opened a new log
I20260925 07:31:58.830031 21219 ts_tablet_manager.cc:1397] T 16707624138c46f69436701eec799546 P 27d46d7f2cbe4b80bd823e1824c4a324: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.000s
I20260925 07:31:58.831916 21219 raft_consensus.cc:348] T 16707624138c46f69436701eec799546 P 27d46d7f2cbe4b80bd823e1824c4a324 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c1b4daf69fe439a9b9fd25e32921420" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 37557 } } peers { permanent_uuid: "27d46d7f2cbe4b80bd823e1824c4a324" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 35135 } } peers { permanent_uuid: "4bb868e1eb6e48ec8edd9c0fe78f1f5f" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 38533 } }
I20260925 07:31:58.832050 21219 raft_consensus.cc:374] T 16707624138c46f69436701eec799546 P 27d46d7f2cbe4b80bd823e1824c4a324 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:31:58.832091 21219 raft_consensus.cc:729] T 16707624138c46f69436701eec799546 P 27d46d7f2cbe4b80bd823e1824c4a324 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27d46d7f2cbe4b80bd823e1824c4a324, State: Initialized, Role: FOLLOWER
I20260925 07:31:58.832190 21219 consensus_queue.cc:260] T 16707624138c46f69436701eec799546 P 27d46d7f2cbe4b80bd823e1824c4a324 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c1b4daf69fe439a9b9fd25e32921420" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 37557 } } peers { permanent_uuid: "27d46d7f2cbe4b80bd823e1824c4a324" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 35135 } } peers { permanent_uuid: "4bb868e1eb6e48ec8edd9c0fe78f1f5f" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 38533 } }
I20260925 07:31:58.832414 21219 ts_tablet_manager.cc:1428] T 16707624138c46f69436701eec799546 P 27d46d7f2cbe4b80bd823e1824c4a324: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 07:31:58.832566 20927 heartbeater.cc:503] Master 127.9.66.62:38097 was elected leader, sending a full tablet report...
Sep 25 07:31:58 dist-test-slave-57fn krb5kdc[20239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321518, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.62@KRBTEST.COM for HTTP/127.9.66.20@KRBTEST.COM
I20260925 07:31:58.883121 21226 raft_consensus.cc:482] T 16707624138c46f69436701eec799546 P 27d46d7f2cbe4b80bd823e1824c4a324 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 07:31:58.883255 21226 raft_consensus.cc:504] T 16707624138c46f69436701eec799546 P 27d46d7f2cbe4b80bd823e1824c4a324 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c1b4daf69fe439a9b9fd25e32921420" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 37557 } } peers { permanent_uuid: "27d46d7f2cbe4b80bd823e1824c4a324" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 35135 } } peers { permanent_uuid: "4bb868e1eb6e48ec8edd9c0fe78f1f5f" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 38533 } }
I20260925 07:31:58.884420 21226 leader_election.cc:288] T 16707624138c46f69436701eec799546 P 27d46d7f2cbe4b80bd823e1824c4a324 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c1b4daf69fe439a9b9fd25e32921420 (127.9.66.2:37557), 4bb868e1eb6e48ec8edd9c0fe78f1f5f (127.9.66.3:38533)
I20260925 07:31:58.905498 21217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:31:58.884011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.1:51679 (local address 127.9.66.2:37557)
0925 07:31:58.884055 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 07:31:58.884057 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:31:58.893715 (+ 9658us) server_negotiation.cc:407] Connection header received
0925 07:31:58.895262 (+ 1547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:31:58.895263 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:31:58.895277 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:31:58.895306 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:31:58.895607 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:31:58.896375 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:31:58.903734 (+ 7359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:31:58.904320 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:31:58.904370 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:31:58.904381 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 07:31:58.905379 (+ 998us) server_negotiation.cc:299] Negotiation successful
0925 07:31:58.905424 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 07:31:58.905776 21228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:31:58.883782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.2:37557 (local address 127.9.66.1:51679)
0925 07:31:58.893622 (+ 9840us) negotiation.cc:107] Waiting for socket to connect
0925 07:31:58.893635 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 07:31:58.895130 (+ 1495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:31:58.895376 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:31:58.895377 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:31:58.895392 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:31:58.895504 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:31:58.895507 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:31:58.896666 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:31:58.896667 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:31:58.898033 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:31:58.898036 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:31:58.904440 (+ 6404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:31:58.904441 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:31:58.904514 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:31:58.904517 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 07:31:58.905657 (+ 1140us) client_negotiation.cc:240] Negotiation successful
0925 07:31:58.905701 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9816,"thread_start_us":112,"threads_started":1}
I20260925 07:31:58.906183 21213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:31:58.884363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.1:55789 (local address 127.9.66.3:38533)
0925 07:31:58.884401 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 07:31:58.884403 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:31:58.896599 (+ 12196us) server_negotiation.cc:407] Connection header received
0925 07:31:58.898191 (+ 1592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:31:58.898192 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:31:58.898202 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:31:58.898230 (+ 28us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:31:58.901604 (+ 3374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:31:58.902332 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:31:58.904599 (+ 2267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:31:58.905114 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:31:58.905158 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:31:58.905167 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 07:31:58.906067 (+ 900us) server_negotiation.cc:299] Negotiation successful
0925 07:31:58.906112 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 07:31:58.906155 21016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16707624138c46f69436701eec799546" candidate_uuid: "27d46d7f2cbe4b80bd823e1824c4a324" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c1b4daf69fe439a9b9fd25e32921420" is_pre_election: true
I20260925 07:31:58.906314 21016 raft_consensus.cc:2454] T 16707624138c46f69436701eec799546 P 0c1b4daf69fe439a9b9fd25e32921420 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27d46d7f2cbe4b80bd823e1824c4a324 in term 0.
I20260925 07:31:58.906562 20815 leader_election.cc:302] T 16707624138c46f69436701eec799546 P 27d46d7f2cbe4b80bd823e1824c4a324 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c1b4daf69fe439a9b9fd25e32921420, 27d46d7f2cbe4b80bd823e1824c4a324; no voters:
I20260925 07:31:58.906697 21226 raft_consensus.cc:2781] T 16707624138c46f69436701eec799546 P 27d46d7f2cbe4b80bd823e1824c4a324 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 07:31:58.906747 21226 raft_consensus.cc:482] T 16707624138c46f69436701eec799546 P 27d46d7f2cbe4b80bd823e1824c4a324 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 07:31:58.906776 21226 raft_consensus.cc:3037] T 16707624138c46f69436701eec799546 P 27d46d7f2cbe4b80bd823e1824c4a324 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:31:58.907711 21226 raft_consensus.cc:504] T 16707624138c46f69436701eec799546 P 27d46d7f2cbe4b80bd823e1824c4a324 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c1b4daf69fe439a9b9fd25e32921420" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 37557 } } peers { permanent_uuid: "27d46d7f2cbe4b80bd823e1824c4a324" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 35135 } } peers { permanent_uuid: "4bb868e1eb6e48ec8edd9c0fe78f1f5f" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 38533 } }
I20260925 07:31:58.908038 21016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16707624138c46f69436701eec799546" candidate_uuid: "27d46d7f2cbe4b80bd823e1824c4a324" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c1b4daf69fe439a9b9fd25e32921420"
I20260925 07:31:58.908130 21016 raft_consensus.cc:3037] T 16707624138c46f69436701eec799546 P 0c1b4daf69fe439a9b9fd25e32921420 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:31:58.909008 21016 raft_consensus.cc:2454] T 16707624138c46f69436701eec799546 P 0c1b4daf69fe439a9b9fd25e32921420 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27d46d7f2cbe4b80bd823e1824c4a324 in term 1.
I20260925 07:31:58.909171 20815 leader_election.cc:302] T 16707624138c46f69436701eec799546 P 27d46d7f2cbe4b80bd823e1824c4a324 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c1b4daf69fe439a9b9fd25e32921420, 27d46d7f2cbe4b80bd823e1824c4a324; no voters:
I20260925 07:31:58.909329 21226 leader_election.cc:288] T 16707624138c46f69436701eec799546 P 27d46d7f2cbe4b80bd823e1824c4a324 [CANDIDATE]: Term 1 election: Requested vote from peers 0c1b4daf69fe439a9b9fd25e32921420 (127.9.66.2:37557), 4bb868e1eb6e48ec8edd9c0fe78f1f5f (127.9.66.3:38533)
I20260925 07:31:58.909377 21226 raft_consensus.cc:2781] T 16707624138c46f69436701eec799546 P 27d46d7f2cbe4b80bd823e1824c4a324 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:31:58.909416 21226 raft_consensus.cc:686] T 16707624138c46f69436701eec799546 P 27d46d7f2cbe4b80bd823e1824c4a324 [term 1 LEADER]: Becoming Leader. State: Replica: 27d46d7f2cbe4b80bd823e1824c4a324, State: Running, Role: LEADER
I20260925 07:31:58.909523 21226 consensus_queue.cc:237] T 16707624138c46f69436701eec799546 P 27d46d7f2cbe4b80bd823e1824c4a324 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c1b4daf69fe439a9b9fd25e32921420" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 37557 } } peers { permanent_uuid: "27d46d7f2cbe4b80bd823e1824c4a324" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 35135 } } peers { permanent_uuid: "4bb868e1eb6e48ec8edd9c0fe78f1f5f" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 38533 } }
I20260925 07:31:58.910297 20715 catalog_manager.cc:5495] T 16707624138c46f69436701eec799546 P 27d46d7f2cbe4b80bd823e1824c4a324 reported cstate change: term changed from 0 to 1, leader changed from <none> to 27d46d7f2cbe4b80bd823e1824c4a324 (127.9.66.1). New cstate: current_term: 1 leader_uuid: "27d46d7f2cbe4b80bd823e1824c4a324" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c1b4daf69fe439a9b9fd25e32921420" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 37557 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27d46d7f2cbe4b80bd823e1824c4a324" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 35135 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4bb868e1eb6e48ec8edd9c0fe78f1f5f" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 38533 } health_report { overall_health: UNKNOWN } } }
I20260925 07:31:58.911510 21227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:31:58.884197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.3:38533 (local address 127.9.66.1:55789)
0925 07:31:58.896496 (+ 12299us) negotiation.cc:107] Waiting for socket to connect
0925 07:31:58.896507 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 07:31:58.898114 (+ 1607us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:31:58.898285 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:31:58.898286 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:31:58.898287 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:31:58.898352 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:31:58.898353 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:31:58.902457 (+ 4104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:31:58.902458 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:31:58.903619 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:31:58.903622 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:31:58.905221 (+ 1599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:31:58.905222 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:31:58.905299 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:31:58.905302 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 07:31:58.906299 (+ 997us) client_negotiation.cc:240] Negotiation successful
0925 07:31:58.906341 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12280,"thread_start_us":67,"threads_started":1}
I20260925 07:31:58.911888 21153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16707624138c46f69436701eec799546" candidate_uuid: "27d46d7f2cbe4b80bd823e1824c4a324" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bb868e1eb6e48ec8edd9c0fe78f1f5f" is_pre_election: true
I20260925 07:31:58.912041 21153 raft_consensus.cc:2454] T 16707624138c46f69436701eec799546 P 4bb868e1eb6e48ec8edd9c0fe78f1f5f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27d46d7f2cbe4b80bd823e1824c4a324 in term 0.
I20260925 07:31:58.912163 21152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16707624138c46f69436701eec799546" candidate_uuid: "27d46d7f2cbe4b80bd823e1824c4a324" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bb868e1eb6e48ec8edd9c0fe78f1f5f"
I20260925 07:31:58.912231 21152 raft_consensus.cc:3037] T 16707624138c46f69436701eec799546 P 4bb868e1eb6e48ec8edd9c0fe78f1f5f [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:31:58.913061 21152 raft_consensus.cc:2454] T 16707624138c46f69436701eec799546 P 4bb868e1eb6e48ec8edd9c0fe78f1f5f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27d46d7f2cbe4b80bd823e1824c4a324 in term 1.
I20260925 07:31:58.949280 21215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:31:58.945896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39144 (local address 127.9.66.1:35135)
0925 07:31:58.945937 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 07:31:58.945938 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 07:31:58.945968 (+ 30us) server_negotiation.cc:407] Connection header received
0925 07:31:58.946061 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:31:58.946062 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:31:58.946084 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:31:58.946149 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 07:31:58.946338 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:31:58.947008 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:31:58.948139 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:31:58.948446 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:31:58.948509 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:31:58.948735 (+ 226us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 07:31:58.948863 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 07:31:58.948943 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 07:31:58.949166 (+ 223us) server_negotiation.cc:299] Negotiation successful
0925 07:31:58.949189 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 07:31:58.953795 21152 raft_consensus.cc:1261] T 16707624138c46f69436701eec799546 P 4bb868e1eb6e48ec8edd9c0fe78f1f5f [term 1 FOLLOWER]: Refusing update from remote peer 27d46d7f2cbe4b80bd823e1824c4a324: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:31:58.954196 21229 consensus_queue.cc:1035] T 16707624138c46f69436701eec799546 P 27d46d7f2cbe4b80bd823e1824c4a324 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bb868e1eb6e48ec8edd9c0fe78f1f5f" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 38533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:31:58.956365 21016 raft_consensus.cc:1261] T 16707624138c46f69436701eec799546 P 0c1b4daf69fe439a9b9fd25e32921420 [term 1 FOLLOWER]: Refusing update from remote peer 27d46d7f2cbe4b80bd823e1824c4a324: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:31:58.956633 21229 consensus_queue.cc:1035] T 16707624138c46f69436701eec799546 P 27d46d7f2cbe4b80bd823e1824c4a324 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c1b4daf69fe439a9b9fd25e32921420" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 37557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:31:58.960942 21232 mvcc.cc:205] Tried to move back new op lower bound from 7333156941631524864 to 7333156941454004224. Current Snapshot: MvccSnapshot[applied={T|T < 7333156941631524864}]
I20260925 07:31:58.961066 21233 mvcc.cc:205] Tried to move back new op lower bound from 7333156941631524864 to 7333156941454004224. Current Snapshot: MvccSnapshot[applied={T|T < 7333156941631524864}]
I20260925 07:31:58.963665 20715 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43964:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 07:31:58.963758 20715 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43964:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 07:31:58.964437 20715 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c9e1263488934ad79d7faf635062ea01: Sending DeleteTablet for 3 replicas of tablet 16707624138c46f69436701eec799546
I20260925 07:31:58.964874 20861 tablet_service.cc:1514] Processing DeleteTablet for tablet 16707624138c46f69436701eec799546 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:31:58 UTC) from {username='kudu', principal='kudu/127.9.66.62@KRBTEST.COM'} at 127.0.0.1:39138
I20260925 07:31:58.965080 21133 tablet_service.cc:1514] Processing DeleteTablet for tablet 16707624138c46f69436701eec799546 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:31:58 UTC) from {username='kudu', principal='kudu/127.9.66.62@KRBTEST.COM'} at 127.0.0.1:54380
I20260925 07:31:58.965389 20999 tablet_service.cc:1514] Processing DeleteTablet for tablet 16707624138c46f69436701eec799546 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:31:58 UTC) from {username='kudu', principal='kudu/127.9.66.62@KRBTEST.COM'} at 127.0.0.1:46772
I20260925 07:31:58.965914 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 20796
I20260925 07:31:58.973454 21241 tablet_replica.cc:330] T 16707624138c46f69436701eec799546 P 4bb868e1eb6e48ec8edd9c0fe78f1f5f: stopping tablet replica
I20260925 07:31:58.973587 21241 raft_consensus.cc:2229] T 16707624138c46f69436701eec799546 P 4bb868e1eb6e48ec8edd9c0fe78f1f5f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 07:31:58.973698 21241 raft_consensus.cc:2258] T 16707624138c46f69436701eec799546 P 4bb868e1eb6e48ec8edd9c0fe78f1f5f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:31:58.973886 21242 tablet_replica.cc:330] T 16707624138c46f69436701eec799546 P 0c1b4daf69fe439a9b9fd25e32921420: stopping tablet replica
I20260925 07:31:58.973981 21242 raft_consensus.cc:2229] T 16707624138c46f69436701eec799546 P 0c1b4daf69fe439a9b9fd25e32921420 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 07:31:58.974040 21242 pending_rounds.cc:62] T 16707624138c46f69436701eec799546 P 0c1b4daf69fe439a9b9fd25e32921420: Trying to abort 2 pending ops.
I20260925 07:31:58.974088 21242 pending_rounds.cc:68] T 16707624138c46f69436701eec799546 P 0c1b4daf69fe439a9b9fd25e32921420: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7333156941631524864 op_type: WRITE_OP write_request { tablet_id: "16707624138c46f69436701eec799546" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "val" type: INT32 is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\003\00090\000\0001\324\000\000" indirect_data: "<redacted>""" } external_consistency_mode: CLIENT_PROPAGATED authz_token { token_data: "\010\232\311\330\325\006\"8\n\ntest-admin\022*\n b0ed3ee32ace4d6ba7ce605e430613e4\020\001\030\001 \001(\001" signature: "<redacted>""(\355\315\376\225\003KL\366\3058\302\371 j\256<5\345\325\315\266n\032TA\317l\341\006\256\031\351/&\254`_\020\ncok?\343v\373\2202\317\270\373\356NY\022s}\323mtU\204\277" signing_key_seq_num: 0 } } request_id { client_id: "6d69e73ee81b461f8a0f9deae2c3e210" seq_no: 0 first_incomplete_seq_no: 0 attempt_no: 0 }
I20260925 07:31:58.974329 21242 pending_rounds.cc:68] T 16707624138c46f69436701eec799546 P 0c1b4daf69fe439a9b9fd25e32921420: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7333156941454004224 op_type: NO_OP noop_request { }
I20260925 07:31:58.974377 21242 raft_consensus.cc:2866] T 16707624138c46f69436701eec799546 P 0c1b4daf69fe439a9b9fd25e32921420 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260925 07:31:58.974402 21242 raft_consensus.cc:2258] T 16707624138c46f69436701eec799546 P 0c1b4daf69fe439a9b9fd25e32921420 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:31:58.974849 21241 ts_tablet_manager.cc:1905] T 16707624138c46f69436701eec799546 P 4bb868e1eb6e48ec8edd9c0fe78f1f5f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 07:31:58.976744 21242 ts_tablet_manager.cc:1905] T 16707624138c46f69436701eec799546 P 0c1b4daf69fe439a9b9fd25e32921420: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 07:31:58.978291 21242 ts_tablet_manager.cc:1918] T 16707624138c46f69436701eec799546 P 0c1b4daf69fe439a9b9fd25e32921420: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 07:31:58.978363 21242 log.cc:1196] T 16707624138c46f69436701eec799546 P 0c1b4daf69fe439a9b9fd25e32921420: Deleting WAL directory at /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/minicluster-data/ts-1/wal/wals/16707624138c46f69436701eec799546
I20260925 07:31:58.978606 21242 ts_tablet_manager.cc:1939] T 16707624138c46f69436701eec799546 P 0c1b4daf69fe439a9b9fd25e32921420: Deleting consensus metadata
I20260925 07:31:58.979045 20700 catalog_manager.cc:4841] TS 0c1b4daf69fe439a9b9fd25e32921420 (127.9.66.2:37557): tablet 16707624138c46f69436701eec799546 (table test-table [id=b0ed3ee32ace4d6ba7ce605e430613e4]) successfully deleted
W20260925 07:31:58.979228 20952 connection.cc:535] server connection from 127.9.66.1:51679 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260925 07:31:58.979465 20701 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.66.1:35135 (error 108)
I20260925 07:31:58.981644 21241 ts_tablet_manager.cc:1918] T 16707624138c46f69436701eec799546 P 4bb868e1eb6e48ec8edd9c0fe78f1f5f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 07:31:58.981742 21241 log.cc:1196] T 16707624138c46f69436701eec799546 P 4bb868e1eb6e48ec8edd9c0fe78f1f5f: Deleting WAL directory at /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790321392278483-9480-0/minicluster-data/ts-2/wal/wals/16707624138c46f69436701eec799546
I20260925 07:31:58.982389 21241 ts_tablet_manager.cc:1939] T 16707624138c46f69436701eec799546 P 4bb868e1eb6e48ec8edd9c0fe78f1f5f: Deleting consensus metadata
I20260925 07:31:58.982853 20702 catalog_manager.cc:4841] TS 4bb868e1eb6e48ec8edd9c0fe78f1f5f (127.9.66.3:38533): tablet 16707624138c46f69436701eec799546 (table test-table [id=b0ed3ee32ace4d6ba7ce605e430613e4]) successfully deleted
I20260925 07:31:58.985754 21212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:31:58.980101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.1:35135 (local address 127.0.0.1:39156)
0925 07:31:58.985637 (+ 5536us) negotiation.cc:107] Waiting for socket to connect
0925 07:31:58.985716 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.1:35135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":5509}
W20260925 07:31:58.985914 20701 catalog_manager.cc:4584] TS 27d46d7f2cbe4b80bd823e1824c4a324 (127.9.66.1:35135): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 16707624138c46f69436701eec799546: Network error: Client connection negotiation failed: client connection to 127.9.66.1:35135: connect: Connection refused (error 111)
I20260925 07:31:58.986326 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 20935
I20260925 07:31:58.991928 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 21070
I20260925 07:31:58.998322 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 20683
I20260925 07:31:59.003090 9480 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260925 07:31:59.347599 9480 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260925 07:31:59.347715 9480 mini_ranger.cc:66] Stopping Ranger...
I20260925 07:31:59.693228 9480 mini_ranger.cc:68] Stopped Ranger
2026-09-25 07:31:59.693 UTC [20269] LOG: received smart shutdown request
2026-09-25 07:31:59.696 UTC [20269] LOG: background worker "logical replication launcher" (PID 20278) exited with exit code 1
2026-09-25 07:31:59.696 UTC [20273] LOG: shutting down
2026-09-25 07:31:59.849 UTC [20269] LOG: database system is shut down
2026-09-25T07:31:59Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (45759 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 07:32:00 dist-test-slave-57fn krb5kdc[21252](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 07:32:00 dist-test-slave-57fn krb5kdc[21252](info): set up 2 sockets
Sep 25 07:32:00 dist-test-slave-57fn krb5kdc[21252](info): commencing operation
krb5kdc: starting...
W20260925 07:32:00.985488 9480 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/krb5kdc/test-user.keytab.
Sep 25 07:32:01 dist-test-slave-57fn krb5kdc[21252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321521, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T07:32:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T07:32:01Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.9.66.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.9.66.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.9.66.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/krb5kdc/rangeradmin_127.9.66.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.9.66.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/krb5kdc/rangeradmin_127.9.66.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.9.66.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.9.66.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.9.66.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/krb5kdc/rangerlookup_127.9.66.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.9.66.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/krb5kdc/rangerlookup_127.9.66.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.9.66.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.66.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.66.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/krb5kdc/HTTP_127.9.66.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.9.66.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/krb5kdc/HTTP_127.9.66.20@KRBTEST.COM.keytab.
I20260925 07:32:01.138295 9480 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task1eZXJS/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/postgres -l logfile start
2026-09-25 07:32:03.480 UTC [21282] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-25 07:32:03.480 UTC [21282] LOG: listening on IPv4 address "127.9.66.20", port 46077
2026-09-25 07:32:03.484 UTC [21282] LOG: listening on Unix socket "/tmp/.s.PGSQL.46077"
2026-09-25 07:32:03.505 UTC [21285] LOG: database system was shut down at 2026-09-25 07:32:02 UTC
2026-09-25 07:32:03.509 UTC [21282] LOG: database system is ready to accept connections
I20260925 07:32:04.457382 9480 mini_postgres.cc:96] Postgres bound to 46077
2026-09-25 07:32:04.463 UTC [21293] FATAL: database "slave" does not exist
127.9.66.20:46077 - accepting connections
I20260925 07:32:04.464394 9480 mini_ranger.cc:161] Starting Ranger...
I20260925 07:32:04.479071 9480 mini_ranger.cc:84] Created miniranger Postgres user
I20260925 07:32:04.942742 9480 mini_ranger.cc:87] Created ranger Postgres database
I20260925 07:32:04.942812 9480 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/ranger-admin
2026-09-25 07:32:05,203 [I] DB FLAVOR :POSTGRES
2026-09-25 07:32:05,204 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 07:32:05,204 [I] Checking connection..
2026-09-25 07:32:05,204 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1eZXJS/build/release/bin/postgresql.jar:/tmp/dist-test-task1eZXJS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.66.20:46077/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-25 07:32:05,455 [I] Checking connection passed.
2026-09-25 07:32:05,455 [I] --------- Verifying version history table ---------
2026-09-25 07:32:05,455 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1eZXJS/build/release/bin/postgresql.jar:/tmp/dist-test-task1eZXJS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.66.20:46077/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 07:32:05,706 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 07:32:05,707 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1eZXJS/build/release/bin/postgresql.jar:/tmp/dist-test-task1eZXJS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.66.20:46077/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 07:32:05,935 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 07:32:05,935 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-25 07:32:05,935 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1eZXJS/build/release/bin/postgresql.jar:/tmp/dist-test-task1eZXJS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.66.20:46077/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task1eZXJS/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-25 07:32:06.154 UTC [21357] WARNING: there is no transaction in progress
2026-09-25 07:32:06,160 [I] create_dbversion_catalog.sql file imported successfully
2026-09-25 07:32:06,160 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1eZXJS/build/release/bin/postgresql.jar:/tmp/dist-test-task1eZXJS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.66.20:46077/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 07:32:06,387 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-25 07:32:06,387 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 07:32:06,388 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1eZXJS/build/release/bin/postgresql.jar:/tmp/dist-test-task1eZXJS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.66.20:46077/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-25 07:32:06,628 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1eZXJS/build/release/bin/postgresql.jar:/tmp/dist-test-task1eZXJS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.66.20:46077/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-25 07:32:06,904 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1eZXJS/build/release/bin/postgresql.jar:/tmp/dist-test-task1eZXJS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.66.20:46077/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-57fn.c.gcp-upstream.internal','N') ;"
2026-09-25 07:32:07,130 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1eZXJS/build/release/bin/postgresql.jar:/tmp/dist-test-task1eZXJS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.66.20:46077/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 07:32:07,363 [I] Table x_portal_user does not exist in database ranger
2026-09-25 07:32:07,364 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1eZXJS/build/release/bin/postgresql.jar:/tmp/dist-test-task1eZXJS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.66.20:46077/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 07:32:07,586 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-25 07:32:07,587 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1eZXJS/build/release/bin/postgresql.jar:/tmp/dist-test-task1eZXJS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.66.20:46077/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 07:32:07,846 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1eZXJS/build/release/bin/postgresql.jar:/tmp/dist-test-task1eZXJS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.66.20:46077/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 07:32:08,067 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-25 07:32:08,068 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1eZXJS/build/release/bin/postgresql.jar:/tmp/dist-test-task1eZXJS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.66.20:46077/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task1eZXJS/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-25 07:32:08.981 UTC [21465] WARNING: there is no transaction in progress
2026-09-25 07:32:08.997 UTC [21465] WARNING: there is no transaction in progress
2026-09-25 07:32:09.013 UTC [21465] WARNING: there is no transaction in progress
2026-09-25 07:32:09.028 UTC [21465] WARNING: there is no transaction in progress
2026-09-25 07:32:09.043 UTC [21465] WARNING: there is no transaction in progress
2026-09-25 07:32:09.060 UTC [21465] WARNING: there is no transaction in progress
2026-09-25 07:32:09.110 UTC [21465] WARNING: there is no transaction in progress
2026-09-25 07:32:09.117 UTC [21465] WARNING: there is no transaction in progress
2026-09-25 07:32:09.131 UTC [21465] WARNING: there is no transaction in progress
2026-09-25 07:32:09.141 UTC [21465] WARNING: there is no transaction in progress
2026-09-25 07:32:09.151 UTC [21465] WARNING: there is no transaction in progress
2026-09-25 07:32:09.164 UTC [21465] WARNING: there is no transaction in progress
2026-09-25 07:32:09.174 UTC [21465] WARNING: there is no transaction in progress
2026-09-25 07:32:09.180 UTC [21465] WARNING: there is no transaction in progress
2026-09-25 07:32:09.186 UTC [21465] WARNING: there is no transaction in progress
2026-09-25 07:32:09.193 UTC [21465] WARNING: there is no transaction in progress
2026-09-25 07:32:09,794 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-25 07:32:09,795 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1eZXJS/build/release/bin/postgresql.jar:/tmp/dist-test-task1eZXJS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.66.20:46077/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-25 07:32:10,024 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-25 07:32:10,024 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1eZXJS/build/release/bin/postgresql.jar:/tmp/dist-test-task1eZXJS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.66.20:46077/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 07:32:10,282 [I] Table x_portal_user already exists in database 'ranger'
2026-09-25 07:32:10,283 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1eZXJS/build/release/bin/postgresql.jar:/tmp/dist-test-task1eZXJS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.66.20:46077/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 07:32:10,540 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-25 07:32:10,540 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1eZXJS/build/release/bin/postgresql.jar:/tmp/dist-test-task1eZXJS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.66.20:46077/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 07:32:10,775 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1eZXJS/build/release/bin/postgresql.jar:/tmp/dist-test-task1eZXJS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.66.20:46077/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 07:32:11,012 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1eZXJS/build/release/bin/postgresql.jar:/tmp/dist-test-task1eZXJS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.66.20:46077/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-57fn.c.gcp-upstream.internal';"
2026-09-25 07:32:11,241 [I] CORE_DB_SCHEMA import status has been updated
2026-09-25 07:32:11,242 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1eZXJS/build/release/bin/postgresql.jar:/tmp/dist-test-task1eZXJS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.66.20:46077/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-25 07:32:11,482 [I] DB_PATCHES have already been applied
I20260925 07:32:11.487705 9480 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/ranger-admin:/tmp/dist-test-task1eZXJS/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task1eZXJS/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task1eZXJS/build/release/bin/postgresql.jar:/tmp/dist-test-task1eZXJS/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260925 07:32:11.487767 9480 mini_ranger.cc:193] Using host: 127.9.66.20
I20260925 07:32:11.490777 9480 mini_ranger.cc:227] Ranger admin URL: http://127.9.66.20:52225
Sep 25, 2026 7:32:11 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 7:32:11 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 7:32:11 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/ranger-admin/ews/webapp
Sep 25, 2026 7:32:11 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 25, 2026 7:32:11 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 25, 2026 7:32:12 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/ranger-admin/ews/webapp].
Sep 25, 2026 7:32:12 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260925 07:32:12.584311 9480 mini_ranger.cc:160] Time spent starting Ranger: real 8.120s user 0.002s sys 0.006s
Sep 25, 2026 7:32:32 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 25, 2026 7:32:32 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.MetricsREST
Sep 25, 2026 7:32:32 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 25, 2026 7:32:32 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 25, 2026 7:32:32 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 25, 2026 7:32:32 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 25, 2026 7:32:32 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 25, 2026 7:32:32 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 25, 2026 7:32:32 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 25, 2026 7:32:32 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 25, 2026 7:32:32 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 25, 2026 7:32:32 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 25, 2026 7:32:32 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 25, 2026 7:32:32 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 25, 2026 7:32:32 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 25, 2026 7:32:32 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 25, 2026 7:32:32 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 25, 2026 7:32:32 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260925 07:32:33.492928 9480 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.9.66.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.9.66.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.9.66.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/krb5kdc/rangerkms_127.9.66.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.9.66.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/krb5kdc/rangerkms_127.9.66.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.9.66.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.9.66.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.66.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/krb5kdc/HTTP_127.9.66.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.9.66.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/krb5kdc/HTTP_127.9.66.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 25 07:32:33 dist-test-slave-57fn krb5kdc[21252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321553, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260925 07:32:33.586053 9480 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260925 07:32:33.595359 9480 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260925 07:32:34.438377 9480 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260925 07:32:34.438444 9480 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/ranger-kms
I20260925 07:32:34.438457 9480 mini_ranger_kms.cc:227] Using postgres at 127.9.66.20:46077
2026-09-25 07:32:34,601 [I] DB FLAVOR :POSTGRES
2026-09-25 07:32:34,601 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 07:32:34,601 [I] Checking connection
2026-09-25 07:32:34,601 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1eZXJS/build/release/bin/postgresql.jar:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.66.20:46077/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-25 07:32:34,845 [I] connection success
2026-09-25 07:32:34,845 [I] --------- Verifying Ranger DB tables ---------
2026-09-25 07:32:34,846 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-25 07:32:34,846 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1eZXJS/build/release/bin/postgresql.jar:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.66.20:46077/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-25 07:32:35,067 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-25 07:32:35,068 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 07:32:35,068 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-25 07:32:35,068 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1eZXJS/build/release/bin/postgresql.jar:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.66.20:46077/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-25 07:32:35,321 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260925 07:32:35.731537 9480 mini_ranger_kms.cc:310] Created kms service
I20260925 07:32:35.899682 9480 mini_ranger_kms.cc:326] Created kudu user
I20260925 07:32:35.940685 9480 mini_ranger_kms.cc:343] Created rangerkms user
Sep 25 07:32:35 dist-test-slave-57fn krb5kdc[21252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321553, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.9.66.20@KRBTEST.COM
I20260925 07:32:36.309417 9480 mini_ranger_kms.cc:384] Added ranger policy
I20260925 07:32:36.309701 9480 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/ranger-kms:/tmp/dist-test-task1eZXJS/build/release/bin/postgresql.jar:/tmp/dist-test-task1eZXJS/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task1eZXJS/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260925 07:32:36.309738 9480 mini_ranger_kms.cc:240] Using host: 127.9.66.20
I20260925 07:32:36.313054 9480 mini_ranger_kms.cc:276] Ranger KMS PID: 21645
I20260925 07:32:36.313128 9480 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.9.66.20:54757
Sep 25, 2026 7:32:37 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 7:32:37 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 7:32:37 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 25, 2026 7:32:37 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 25, 2026 7:32:37 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/ranger-kms/ews/webapp].
Sep 25, 2026 7:32:37 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 25 07:32:38 dist-test-slave-57fn krb5kdc[21252](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790321558, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.9.66.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:32:38.509948 9480 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.924s user 0.012s sys 0.132s
I20260925 07:32:38.510040 9480 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260925 07:32:38.510085 9480 mini_ranger_kms.cc:401] 127.9.66.20:54757/kms/v1/keys
Sep 25 07:32:38 dist-test-slave-57fn krb5kdc[21252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321558, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.9.66.20@KRBTEST.COM for HTTP/127.9.66.20@KRBTEST.COM
Sep 25 07:32:39 dist-test-slave-57fn krb5kdc[21252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321558, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.9.66.20@KRBTEST.COM for HTTP/127.9.66.20@KRBTEST.COM
Sep 25 07:32:41 dist-test-slave-57fn krb5kdc[21252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321561, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.9.66.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.66.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.66.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab.
Entry for principal kudu/127.9.66.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab.
I20260925 07:32:41.397450 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.66.62:35385
--webserver_interface=127.9.66.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab
--principal=kudu/127.9.66.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:46093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.66.62:35385
--ranger_config_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.66.20:54757/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:32:41.484601 21696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:32:41.484815 21696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:32:41.484838 21696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:32:41.486271 21696 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:32:41.486328 21696 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:32:41.486344 21696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:32:41.486357 21696 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:32:41.486371 21696 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:32:41.488102 21696 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:46093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/master-0/wal
--ranger_kms_url=127.9.66.20:54757/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.9.66.62:35385
--ranger_config_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab
--principal=kudu/127.9.66.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.66.62:35385
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.66.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21696
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:32:41.488354 21696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:32:41.488617 21696 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:32:41.493944 21702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:32:41.493963 21701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:32:41.494239 21696 server_base.cc:1034] running on GCE node
W20260925 07:32:41.493952 21704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:32:41.494588 21696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:32:41.494925 21696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:32:41.496101 21696 hybrid_clock.cc:648] HybridClock initialized: now 1790321561496077 us; error 41 us; skew 500 ppm
Sep 25 07:32:41 dist-test-slave-57fn krb5kdc[21252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321561, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:32:41.498929 21696 init.cc:377] Logged in from keytab as kudu/127.9.66.62@KRBTEST.COM (short username kudu)
I20260925 07:32:41.499416 21696 webserver.cc:460] Webserver started at http://127.9.66.62:44999/ using document root <none> and password file <none>
I20260925 07:32:41.499641 21696 fs_manager.cc:359] Metadata directory not provided
I20260925 07:32:41.499701 21696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:32:41.499898 21696 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 07:32:41 dist-test-slave-57fn krb5kdc[21252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321561, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.62@KRBTEST.COM for HTTP/127.9.66.20@KRBTEST.COM
I20260925 07:32:41.553258 21696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/master-0/data/instance:
uuid: "d7c638cf5b1545c787782729be7883b5"
format_stamp: "Formatted at 2026-09-25 07:32:41 on dist-test-slave-57fn"
server_key: "b7b86bc7916e283fcecb44dfd97eb5ae"
server_key_iv: "cf872f285877daf3f74b41d7fd145e75"
server_key_version: "kuduclusterkey@0"
I20260925 07:32:41.553818 21696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/master-0/wal/instance:
uuid: "d7c638cf5b1545c787782729be7883b5"
format_stamp: "Formatted at 2026-09-25 07:32:41 on dist-test-slave-57fn"
server_key: "b7b86bc7916e283fcecb44dfd97eb5ae"
server_key_iv: "cf872f285877daf3f74b41d7fd145e75"
server_key_version: "kuduclusterkey@0"
I20260925 07:32:41.555660 21696 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260925 07:32:41.570242 21711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:32:41.570493 21696 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 07:32:41.570600 21696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/master-0/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/master-0/wal
uuid: "d7c638cf5b1545c787782729be7883b5"
format_stamp: "Formatted at 2026-09-25 07:32:41 on dist-test-slave-57fn"
server_key: "b7b86bc7916e283fcecb44dfd97eb5ae"
server_key_iv: "cf872f285877daf3f74b41d7fd145e75"
server_key_version: "kuduclusterkey@0"
I20260925 07:32:41.570703 21696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:32:41.600776 21696 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:32:41.602960 21696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:32:41.603117 21696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:32:41.608204 21696 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.62:35385
I20260925 07:32:41.608289 21763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.62:35385 every 8 connection(s)
I20260925 07:32:41.608664 21696 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
I20260925 07:32:41.609383 21764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:32:41.611685 21764 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7c638cf5b1545c787782729be7883b5: Bootstrap starting.
I20260925 07:32:41.612273 21764 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d7c638cf5b1545c787782729be7883b5: Neither blocks nor log segments found. Creating new log.
I20260925 07:32:41.612555 21764 log.cc:824] T 00000000000000000000000000000000 P d7c638cf5b1545c787782729be7883b5: Log is configured to *not* fsync() on all Append() calls
I20260925 07:32:41.613261 21764 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7c638cf5b1545c787782729be7883b5: No bootstrap required, opened a new log
I20260925 07:32:41.613677 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 21696
I20260925 07:32:41.613773 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/master-0/wal/instance
I20260925 07:32:41.613961 9480 external_mini_cluster.cc:1428] Setting key 9d9241edbb440215e4e16ef5f3549f84
I20260925 07:32:41.615736 21764 raft_consensus.cc:348] T 00000000000000000000000000000000 P d7c638cf5b1545c787782729be7883b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7c638cf5b1545c787782729be7883b5" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 35385 } }
I20260925 07:32:41.615890 21764 raft_consensus.cc:374] T 00000000000000000000000000000000 P d7c638cf5b1545c787782729be7883b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:32:41.615952 21764 raft_consensus.cc:729] T 00000000000000000000000000000000 P d7c638cf5b1545c787782729be7883b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7c638cf5b1545c787782729be7883b5, State: Initialized, Role: FOLLOWER
I20260925 07:32:41.616089 21764 consensus_queue.cc:260] T 00000000000000000000000000000000 P d7c638cf5b1545c787782729be7883b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7c638cf5b1545c787782729be7883b5" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 35385 } }
I20260925 07:32:41.616175 21764 raft_consensus.cc:388] T 00000000000000000000000000000000 P d7c638cf5b1545c787782729be7883b5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 07:32:41.616218 21764 raft_consensus.cc:482] T 00000000000000000000000000000000 P d7c638cf5b1545c787782729be7883b5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 07:32:41.616287 21764 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d7c638cf5b1545c787782729be7883b5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:32:41.616890 21764 raft_consensus.cc:504] T 00000000000000000000000000000000 P d7c638cf5b1545c787782729be7883b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7c638cf5b1545c787782729be7883b5" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 35385 } }
I20260925 07:32:41.616999 21764 leader_election.cc:302] T 00000000000000000000000000000000 P d7c638cf5b1545c787782729be7883b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d7c638cf5b1545c787782729be7883b5; no voters:
I20260925 07:32:41.617193 21764 leader_election.cc:288] T 00000000000000000000000000000000 P d7c638cf5b1545c787782729be7883b5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 07:32:41.617230 21769 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d7c638cf5b1545c787782729be7883b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:32:41.617333 21769 raft_consensus.cc:686] T 00000000000000000000000000000000 P d7c638cf5b1545c787782729be7883b5 [term 1 LEADER]: Becoming Leader. State: Replica: d7c638cf5b1545c787782729be7883b5, State: Running, Role: LEADER
I20260925 07:32:41.617441 21764 sys_catalog.cc:564] T 00000000000000000000000000000000 P d7c638cf5b1545c787782729be7883b5 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:32:41.617475 21769 consensus_queue.cc:237] T 00000000000000000000000000000000 P d7c638cf5b1545c787782729be7883b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7c638cf5b1545c787782729be7883b5" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 35385 } }
I20260925 07:32:41.617923 21770 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7c638cf5b1545c787782729be7883b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d7c638cf5b1545c787782729be7883b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7c638cf5b1545c787782729be7883b5" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 35385 } } }
I20260925 07:32:41.617942 21771 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7c638cf5b1545c787782729be7883b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d7c638cf5b1545c787782729be7883b5. Latest consensus state: current_term: 1 leader_uuid: "d7c638cf5b1545c787782729be7883b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7c638cf5b1545c787782729be7883b5" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 35385 } } }
I20260925 07:32:41.618029 21770 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7c638cf5b1545c787782729be7883b5 [sys.catalog]: This master's current role is: LEADER
I20260925 07:32:41.618048 21771 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7c638cf5b1545c787782729be7883b5 [sys.catalog]: This master's current role is: LEADER
I20260925 07:32:41.618244 21777 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 07:32:41.618831 21777 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 07:32:41.619087 21764 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
Sep 25 07:32:41 dist-test-slave-57fn krb5kdc[21252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321561, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.66.62@KRBTEST.COM
I20260925 07:32:41.624362 21777 catalog_manager.cc:1347] Generated new cluster ID: e7d9ed68c344403b903daa170320b951
I20260925 07:32:41.624433 21777 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 07:32:41.628325 21768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:41.614493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51502 (local address 127.9.66.62:35385)
0925 07:32:41.614675 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 07:32:41.614691 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 07:32:41.614743 (+ 52us) server_negotiation.cc:407] Connection header received
0925 07:32:41.615731 (+ 988us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:32:41.615738 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:32:41.615786 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:32:41.615946 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:32:41.616713 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:32:41.617913 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:32:41.619029 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:32:41.621489 (+ 2460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:32:41.622041 (+ 552us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:32:41.624412 (+ 2371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:32:41.624417 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:32:41.624417 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:32:41.624430 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:32:41.626417 (+ 1987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:32:41.626808 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:32:41.626810 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:32:41.626811 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:32:41.626926 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:32:41.627119 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:32:41.627120 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:32:41.627120 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:32:41.627396 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:32:41.627516 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0925 07:32:41.627975 (+ 459us) server_negotiation.cc:299] Negotiation successful
0925 07:32:41.628109 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":95,"threads_started":1}
I20260925 07:32:41.635912 21777 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 07:32:41.636646 21777 catalog_manager.cc:1504] Loading token signing keys...
I20260925 07:32:41.642139 21777 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d7c638cf5b1545c787782729be7883b5: Generated new TSK 0
I20260925 07:32:41.642308 21777 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.66.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.66.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.66.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.1.keytab.
Entry for principal kudu/127.9.66.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.1.keytab.
I20260925 07:32:42.318698 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.66.20:54757/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.66.1:0
--local_ip_for_outbound_sockets=127.9.66.1
--webserver_interface=127.9.66.1
--webserver_port=0
--tserver_master_addrs=127.9.66.62:35385
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.1.keytab
--principal=kudu/127.9.66.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:46093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:32:42.427351 21809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:32:42.427548 21809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:32:42.427572 21809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:32:42.428735 21809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:32:42.428776 21809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:32:42.428800 21809 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.66.1
I20260925 07:32:42.430361 21809 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:46093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.9.66.20:54757/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.1.keytab
--principal=kudu/127.9.66.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.66.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.66.1
--webserver_port=0
--tserver_master_addrs=127.9.66.62:35385
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21809
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.66.1
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:32:42.430583 21809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:32:42.430819 21809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:32:42.436610 21815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:32:42.436615 21814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:32:42.436620 21817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:32:42.436890 21809 server_base.cc:1034] running on GCE node
I20260925 07:32:42.437091 21809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:32:42.437366 21809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:32:42.440653 21809 hybrid_clock.cc:648] HybridClock initialized: now 1790321562440630 us; error 31 us; skew 500 ppm
Sep 25 07:32:42 dist-test-slave-57fn krb5kdc[21252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321562, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:32:42.443883 21809 init.cc:377] Logged in from keytab as kudu/127.9.66.1@KRBTEST.COM (short username kudu)
I20260925 07:32:42.444394 21809 webserver.cc:460] Webserver started at http://127.9.66.1:34123/ using document root <none> and password file <none>
I20260925 07:32:42.444703 21809 fs_manager.cc:359] Metadata directory not provided
I20260925 07:32:42.444769 21809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:32:42.444886 21809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 07:32:42 dist-test-slave-57fn krb5kdc[21252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321562, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.1@KRBTEST.COM for HTTP/127.9.66.20@KRBTEST.COM
I20260925 07:32:42.459586 21809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/ts-0/data/instance:
uuid: "528e95197ba948cf9fbb1cdfae5f3def"
format_stamp: "Formatted at 2026-09-25 07:32:42 on dist-test-slave-57fn"
server_key: "294d6fd2de939bcba60e21f9fcc7a1cf"
server_key_iv: "b059387ffb3dee1aa8ddfa148e6c611e"
server_key_version: "kuduclusterkey@0"
I20260925 07:32:42.460070 21809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/ts-0/wal/instance:
uuid: "528e95197ba948cf9fbb1cdfae5f3def"
format_stamp: "Formatted at 2026-09-25 07:32:42 on dist-test-slave-57fn"
server_key: "294d6fd2de939bcba60e21f9fcc7a1cf"
server_key_iv: "b059387ffb3dee1aa8ddfa148e6c611e"
server_key_version: "kuduclusterkey@0"
I20260925 07:32:42.464768 21809 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260925 07:32:42.481012 21826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:32:42.481236 21809 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260925 07:32:42.481310 21809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/ts-0/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/ts-0/wal
uuid: "528e95197ba948cf9fbb1cdfae5f3def"
format_stamp: "Formatted at 2026-09-25 07:32:42 on dist-test-slave-57fn"
server_key: "294d6fd2de939bcba60e21f9fcc7a1cf"
server_key_iv: "b059387ffb3dee1aa8ddfa148e6c611e"
server_key_version: "kuduclusterkey@0"
I20260925 07:32:42.481390 21809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:32:42.509500 21809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:32:42.512007 21809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:32:42.512197 21809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:32:42.512481 21809 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:32:42.512878 21809 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:32:42.512974 21809 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:32:42.513091 21809 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:32:42.513159 21809 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:32:42.519788 21809 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.1:46341
I20260925 07:32:42.520224 21809 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/ts-0/data/info.pb
I20260925 07:32:42.523458 21939 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.1:46341 every 8 connection(s)
I20260925 07:32:42.525523 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 21809
I20260925 07:32:42.525655 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/ts-0/wal/instance
I20260925 07:32:42.525815 9480 external_mini_cluster.cc:1428] Setting key 036745f8f4b9b1e18c240bd3d6ed8be5
Sep 25 07:32:42 dist-test-slave-57fn krb5kdc[21252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321562, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.1@KRBTEST.COM for kudu/127.9.66.62@KRBTEST.COM
I20260925 07:32:42.538338 21942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:42.524091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:35385 (local address 127.9.66.1:38317)
0925 07:32:42.524565 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0925 07:32:42.524612 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 07:32:42.525200 (+ 588us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:32:42.526421 (+ 1221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:32:42.526425 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:32:42.528229 (+ 1804us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:32:42.528504 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:32:42.528510 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:32:42.529448 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:32:42.529449 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:32:42.530795 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:32:42.530798 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:32:42.531351 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:32:42.531352 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:32:42.531492 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:32:42.532075 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:32:42.532085 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:32:42.533730 (+ 1645us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:32:42.535733 (+ 2003us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:32:42.535739 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:32:42.535742 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:32:42.536178 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:32:42.536421 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:32:42.536421 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:32:42.536421 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:32:42.536543 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:32:42.537606 (+ 1063us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:32:42.537613 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:32:42.537798 (+ 185us) client_negotiation.cc:769] Sending connection context
0925 07:32:42.537928 (+ 130us) client_negotiation.cc:240] Negotiation successful
0925 07:32:42.538099 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":77,"threads_started":1}
I20260925 07:32:42.538427 21943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:42.524380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.1:38317 (local address 127.9.66.62:35385)
0925 07:32:42.526178 (+ 1798us) server_negotiation.cc:206] Beginning negotiation
0925 07:32:42.526180 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:32:42.526189 (+ 9us) server_negotiation.cc:407] Connection header received
0925 07:32:42.526213 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:32:42.526214 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:32:42.526225 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:32:42.526292 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:32:42.528605 (+ 2313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:32:42.529332 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:32:42.530895 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:32:42.531208 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:32:42.531267 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:32:42.533827 (+ 2560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:32:42.533828 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:32:42.533829 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:32:42.533836 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:32:42.535642 (+ 1806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:32:42.536275 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:32:42.536289 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:32:42.536289 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:32:42.536357 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:32:42.536628 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:32:42.536629 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:32:42.536629 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:32:42.536761 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:32:42.536822 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 07:32:42.538222 (+ 1400us) server_negotiation.cc:299] Negotiation successful
0925 07:32:42.538307 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1771,"thread_start_us":84,"threads_started":1}
I20260925 07:32:42.538986 21940 heartbeater.cc:347] Connected to a master server at 127.9.66.62:35385
I20260925 07:32:42.539094 21940 heartbeater.cc:464] Registering TS with master...
I20260925 07:32:42.539283 21940 heartbeater.cc:511] Master 127.9.66.62:35385 requested a full tablet report, sending...
I20260925 07:32:42.539836 21728 ts_manager.cc:194] Registered new tserver with Master: 528e95197ba948cf9fbb1cdfae5f3def (127.9.66.1:46341)
I20260925 07:32:42.540779 21728 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.66.1@KRBTEST.COM'} at 127.9.66.1:38317
WARNING: no policy specified for kudu/127.9.66.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.66.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.66.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.2.keytab.
Entry for principal kudu/127.9.66.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.2.keytab.
I20260925 07:32:42.564826 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.66.20:54757/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.66.2:0
--local_ip_for_outbound_sockets=127.9.66.2
--webserver_interface=127.9.66.2
--webserver_port=0
--tserver_master_addrs=127.9.66.62:35385
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.2.keytab
--principal=kudu/127.9.66.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:46093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 07:32:42 dist-test-slave-57fn krb5kdc[21252](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790321562, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260925 07:32:42.672889 21948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:32:42.673096 21948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:32:42.673117 21948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:32:42.674350 21948 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:32:42.674393 21948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:32:42.674444 21948 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.66.2
I20260925 07:32:42.676111 21948 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:46093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.9.66.20:54757/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.2.keytab
--principal=kudu/127.9.66.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.66.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.66.2
--webserver_port=0
--tserver_master_addrs=127.9.66.62:35385
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21948
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.66.2
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:32:42.676357 21948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:32:42.676578 21948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:32:42.682503 21954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:32:42.682490 21956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:32:42.682544 21953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:32:42.682796 21948 server_base.cc:1034] running on GCE node
I20260925 07:32:42.682977 21948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:32:42.683231 21948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:32:42.684405 21948 hybrid_clock.cc:648] HybridClock initialized: now 1790321562684142 us; error 277 us; skew 500 ppm
Sep 25 07:32:42 dist-test-slave-57fn krb5kdc[21252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321562, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:32:42.687243 21948 init.cc:377] Logged in from keytab as kudu/127.9.66.2@KRBTEST.COM (short username kudu)
I20260925 07:32:42.687747 21948 webserver.cc:460] Webserver started at http://127.9.66.2:43867/ using document root <none> and password file <none>
I20260925 07:32:42.688004 21948 fs_manager.cc:359] Metadata directory not provided
I20260925 07:32:42.688086 21948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:32:42.688215 21948 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 07:32:42 dist-test-slave-57fn krb5kdc[21252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321562, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.2@KRBTEST.COM for HTTP/127.9.66.20@KRBTEST.COM
I20260925 07:32:42.699702 21948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/ts-1/data/instance:
uuid: "02d6b2ab7add458193cf60d80f814b44"
format_stamp: "Formatted at 2026-09-25 07:32:42 on dist-test-slave-57fn"
server_key: "fc500c1e801e01978d2e7dd43cf77c31"
server_key_iv: "2e2d3735e8b9ee5fb9881d2c74d2a104"
server_key_version: "kuduclusterkey@0"
I20260925 07:32:42.700084 21948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/ts-1/wal/instance:
uuid: "02d6b2ab7add458193cf60d80f814b44"
format_stamp: "Formatted at 2026-09-25 07:32:42 on dist-test-slave-57fn"
server_key: "fc500c1e801e01978d2e7dd43cf77c31"
server_key_iv: "2e2d3735e8b9ee5fb9881d2c74d2a104"
server_key_version: "kuduclusterkey@0"
I20260925 07:32:42.701483 21948 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 07:32:42.719169 21964 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:32:42.719338 21948 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.000s sys 0.000s
I20260925 07:32:42.719434 21948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/ts-1/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/ts-1/wal
uuid: "02d6b2ab7add458193cf60d80f814b44"
format_stamp: "Formatted at 2026-09-25 07:32:42 on dist-test-slave-57fn"
server_key: "fc500c1e801e01978d2e7dd43cf77c31"
server_key_iv: "2e2d3735e8b9ee5fb9881d2c74d2a104"
server_key_version: "kuduclusterkey@0"
I20260925 07:32:42.719530 21948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:32:42.738690 21948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:32:42.741016 21948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:32:42.741185 21948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:32:42.741379 21948 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:32:42.741748 21948 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:32:42.741797 21948 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:32:42.741834 21948 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:32:42.741858 21948 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:32:42.748111 21948 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.2:36895
I20260925 07:32:42.748545 21948 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/ts-1/data/info.pb
I20260925 07:32:42.753986 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 21948
I20260925 07:32:42.754086 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/ts-1/wal/instance
I20260925 07:32:42.754254 9480 external_mini_cluster.cc:1428] Setting key d67a2634aa342bbda70457fe16dd561b
I20260925 07:32:42.750988 22077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.2:36895 every 8 connection(s)
WARNING: no policy specified for kudu/127.9.66.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.66.3@KRBTEST.COM" created.
Sep 25 07:32:42 dist-test-slave-57fn krb5kdc[21252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321562, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.2@KRBTEST.COM for kudu/127.9.66.62@KRBTEST.COM
I20260925 07:32:42.774984 21943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:42.751669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.2:54425 (local address 127.9.66.62:35385)
0925 07:32:42.751917 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0925 07:32:42.751919 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:32:42.754389 (+ 2470us) server_negotiation.cc:407] Connection header received
0925 07:32:42.755376 (+ 987us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:32:42.755377 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:32:42.755387 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:32:42.755416 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:32:42.756256 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:32:42.757032 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:32:42.758544 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:32:42.758834 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:32:42.758900 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:32:42.771083 (+ 12183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:32:42.771085 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:32:42.771086 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:32:42.771094 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:32:42.773062 (+ 1968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:32:42.773714 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:32:42.773716 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:32:42.773716 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:32:42.773777 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:32:42.774107 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:32:42.774108 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:32:42.774108 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:32:42.774291 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:32:42.774350 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 07:32:42.774759 (+ 409us) server_negotiation.cc:299] Negotiation successful
0925 07:32:42.774872 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214}
I20260925 07:32:42.775560 22080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:42.751635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:35385 (local address 127.9.66.2:54425)
0925 07:32:42.754249 (+ 2614us) negotiation.cc:107] Waiting for socket to connect
0925 07:32:42.754297 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 07:32:42.755250 (+ 953us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:32:42.755595 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:32:42.755600 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:32:42.755869 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:32:42.756142 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:32:42.756150 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:32:42.757167 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:32:42.757168 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:32:42.758436 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:32:42.758439 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:32:42.759027 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:32:42.759028 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:32:42.759178 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:32:42.759759 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:32:42.759769 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:32:42.770946 (+ 11177us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:32:42.773227 (+ 2281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:32:42.773238 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:32:42.773242 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:32:42.773614 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:32:42.773865 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:32:42.773866 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:32:42.773866 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:32:42.774023 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:32:42.774387 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:32:42.774395 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:32:42.774594 (+ 199us) client_negotiation.cc:769] Sending connection context
0925 07:32:42.775125 (+ 531us) client_negotiation.cc:240] Negotiation successful
0925 07:32:42.775307 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2569,"thread_start_us":99,"threads_started":1}
I20260925 07:32:42.776037 22078 heartbeater.cc:347] Connected to a master server at 127.9.66.62:35385
I20260925 07:32:42.776139 22078 heartbeater.cc:464] Registering TS with master...
I20260925 07:32:42.776350 22078 heartbeater.cc:511] Master 127.9.66.62:35385 requested a full tablet report, sending...
I20260925 07:32:42.776692 21728 ts_manager.cc:194] Registered new tserver with Master: 02d6b2ab7add458193cf60d80f814b44 (127.9.66.2:36895)
I20260925 07:32:42.777206 21728 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.66.2@KRBTEST.COM'} at 127.9.66.2:54425
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.66.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.3.keytab.
Entry for principal kudu/127.9.66.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.3.keytab.
I20260925 07:32:42.784638 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.66.20:54757/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.66.3:0
--local_ip_for_outbound_sockets=127.9.66.3
--webserver_interface=127.9.66.3
--webserver_port=0
--tserver_master_addrs=127.9.66.62:35385
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.3.keytab
--principal=kudu/127.9.66.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:46093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:32:42.899698 22083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:32:42.900027 22083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:32:42.900115 22083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:32:42.901793 22083 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:32:42.901891 22083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:32:42.901968 22083 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.66.3
I20260925 07:32:42.904129 22083 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:46093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.9.66.20:54757/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.3.keytab
--principal=kudu/127.9.66.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.66.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.66.3
--webserver_port=0
--tserver_master_addrs=127.9.66.62:35385
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22083
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.66.3
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:32:42.904587 22083 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:32:42.904909 22083 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:32:42.911201 22089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:32:42.911208 22091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:32:42.911384 22088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:32:42.911823 22083 server_base.cc:1034] running on GCE node
I20260925 07:32:42.912029 22083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:32:42.912267 22083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:32:42.913436 22083 hybrid_clock.cc:648] HybridClock initialized: now 1790321562913411 us; error 41 us; skew 500 ppm
Sep 25 07:32:42 dist-test-slave-57fn krb5kdc[21252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321562, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:32:42.916508 22083 init.cc:377] Logged in from keytab as kudu/127.9.66.3@KRBTEST.COM (short username kudu)
I20260925 07:32:42.917012 22083 webserver.cc:460] Webserver started at http://127.9.66.3:44015/ using document root <none> and password file <none>
I20260925 07:32:42.917280 22083 fs_manager.cc:359] Metadata directory not provided
I20260925 07:32:42.917335 22083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:32:42.917457 22083 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 07:32:42 dist-test-slave-57fn krb5kdc[21252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321562, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.3@KRBTEST.COM for HTTP/127.9.66.20@KRBTEST.COM
I20260925 07:32:42.931442 22083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/ts-2/data/instance:
uuid: "fa14e49f1dd243b6aa588d65fabe6c9a"
format_stamp: "Formatted at 2026-09-25 07:32:42 on dist-test-slave-57fn"
server_key: "55b945c1674a4707a053df64f5305780"
server_key_iv: "2f3cd8e30f1b39d263bd0d8c280682ff"
server_key_version: "kuduclusterkey@0"
I20260925 07:32:42.931978 22083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/ts-2/wal/instance:
uuid: "fa14e49f1dd243b6aa588d65fabe6c9a"
format_stamp: "Formatted at 2026-09-25 07:32:42 on dist-test-slave-57fn"
server_key: "55b945c1674a4707a053df64f5305780"
server_key_iv: "2f3cd8e30f1b39d263bd0d8c280682ff"
server_key_version: "kuduclusterkey@0"
I20260925 07:32:42.934034 22083 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260925 07:32:42.947902 22098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:32:42.948057 22083 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:32:42.948130 22083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/ts-2/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/ts-2/wal
uuid: "fa14e49f1dd243b6aa588d65fabe6c9a"
format_stamp: "Formatted at 2026-09-25 07:32:42 on dist-test-slave-57fn"
server_key: "55b945c1674a4707a053df64f5305780"
server_key_iv: "2f3cd8e30f1b39d263bd0d8c280682ff"
server_key_version: "kuduclusterkey@0"
I20260925 07:32:42.948199 22083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:32:42.961104 22083 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:32:42.963330 22083 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:32:42.963465 22083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:32:42.963660 22083 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:32:42.963975 22083 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:32:42.964025 22083 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:32:42.964072 22083 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:32:42.964094 22083 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:32:42.971158 22083 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.3:33415
I20260925 07:32:42.971570 22083 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/ts-2/data/info.pb
I20260925 07:32:42.973318 22211 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.3:33415 every 8 connection(s)
Sep 25 07:32:42 dist-test-slave-57fn krb5kdc[21252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321562, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.3@KRBTEST.COM for kudu/127.9.66.62@KRBTEST.COM
I20260925 07:32:42.982826 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 22083
I20260925 07:32:42.982928 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/ts-2/wal/instance
I20260925 07:32:42.983088 9480 external_mini_cluster.cc:1428] Setting key 7f936feb4d606d2d8a79f54edf1a7daa
I20260925 07:32:42.992448 21943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:42.974130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.3:41547 (local address 127.9.66.62:35385)
0925 07:32:42.974189 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 07:32:42.974192 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:32:42.974444 (+ 252us) server_negotiation.cc:407] Connection header received
0925 07:32:42.975038 (+ 594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:32:42.975039 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:32:42.975052 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:32:42.975102 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:32:42.975748 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:32:42.976375 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:32:42.977352 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:32:42.977646 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:32:42.977710 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:32:42.981371 (+ 3661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:32:42.981372 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:32:42.981373 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:32:42.981380 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:32:42.983281 (+ 1901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:32:42.990041 (+ 6760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:32:42.990042 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:32:42.990043 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:32:42.990103 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:32:42.991714 (+ 1611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:32:42.991715 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:32:42.991715 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:32:42.991853 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:32:42.991894 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0925 07:32:42.992265 (+ 371us) server_negotiation.cc:299] Negotiation successful
0925 07:32:42.992363 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 07:32:42.993378 22214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:42.973873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:35385 (local address 127.9.66.3:41547)
0925 07:32:42.974302 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0925 07:32:42.974335 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 07:32:42.974922 (+ 587us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:32:42.975170 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:32:42.975174 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:32:42.975399 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:32:42.975649 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:32:42.975654 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:32:42.976468 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:32:42.976469 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:32:42.977249 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:32:42.977251 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:32:42.977788 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:32:42.977789 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:32:42.977934 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:32:42.978532 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:32:42.978542 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:32:42.981259 (+ 2717us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:32:42.989634 (+ 8375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:32:42.989641 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:32:42.989644 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:32:42.989950 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:32:42.991464 (+ 1514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:32:42.991465 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:32:42.991466 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:32:42.991623 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:32:42.991938 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:32:42.991944 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:32:42.992116 (+ 172us) client_negotiation.cc:769] Sending connection context
0925 07:32:42.992793 (+ 677us) client_negotiation.cc:240] Negotiation successful
0925 07:32:42.992954 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":72,"threads_started":1}
I20260925 07:32:42.994112 22212 heartbeater.cc:347] Connected to a master server at 127.9.66.62:35385
I20260925 07:32:42.994200 22212 heartbeater.cc:464] Registering TS with master...
I20260925 07:32:42.994367 22212 heartbeater.cc:511] Master 127.9.66.62:35385 requested a full tablet report, sending...
I20260925 07:32:42.995211 21728 ts_manager.cc:194] Registered new tserver with Master: fa14e49f1dd243b6aa588d65fabe6c9a (127.9.66.3:33415)
I20260925 07:32:42.995689 21728 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.66.3@KRBTEST.COM'} at 127.9.66.3:41547
I20260925 07:32:43.005666 9480 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 07:32:43.006466 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 21809
I20260925 07:32:43.013742 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 21948
I20260925 07:32:43.020670 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 22083
I20260925 07:32:43.026868 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 21696
I20260925 07:32:43.032176 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.66.62:35385
--webserver_interface=127.9.66.62
--webserver_port=44999
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab
--principal=kudu/127.9.66.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:46093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.66.62:35385
--ranger_config_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.66.20:54757/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.9.66.20:54757/kms with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:32:43.111560 22215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:32:43.111761 22215 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:32:43.111783 22215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:32:43.112919 22215 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:32:43.112960 22215 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:32:43.112974 22215 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:32:43.112987 22215 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:32:43.112999 22215 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:32:43.114578 22215 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:46093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.9.66.20:54757/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.9.66.62:35385
--ranger_config_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab
--principal=kudu/127.9.66.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.66.62:35385
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.66.62
--webserver_port=44999
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22215
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:32:43.114795 22215 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:32:43.115042 22215 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:32:43.120735 22221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:32:43.120766 22220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:32:43.120862 22223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:32:43.121033 22215 server_base.cc:1034] running on GCE node
I20260925 07:32:43.121312 22215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:32:43.121685 22215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:32:43.122854 22215 hybrid_clock.cc:648] HybridClock initialized: now 1790321563122845 us; error 42 us; skew 500 ppm
Sep 25 07:32:43 dist-test-slave-57fn krb5kdc[21252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321563, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:32:43.125312 22215 init.cc:377] Logged in from keytab as kudu/127.9.66.62@KRBTEST.COM (short username kudu)
I20260925 07:32:43.125800 22215 webserver.cc:460] Webserver started at http://127.9.66.62:44999/ using document root <none> and password file <none>
I20260925 07:32:43.126044 22215 fs_manager.cc:359] Metadata directory not provided
I20260925 07:32:43.126094 22215 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 07:32:43 dist-test-slave-57fn krb5kdc[21252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321563, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.62@KRBTEST.COM for HTTP/127.9.66.20@KRBTEST.COM
I20260925 07:32:43.169471 22215 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:32:43.170262 22231 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:32:43.170477 22215 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:32:43.170575 22215 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/master-0/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/master-0/wal
uuid: "d7c638cf5b1545c787782729be7883b5"
format_stamp: "Formatted at 2026-09-25 07:32:41 on dist-test-slave-57fn"
server_key: "b7b86bc7916e283fcecb44dfd97eb5ae"
server_key_iv: "cf872f285877daf3f74b41d7fd145e75"
server_key_version: "kuduclusterkey@0"
I20260925 07:32:43.170949 22215 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:32:43.197207 22215 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:32:43.199436 22215 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:32:43.199604 22215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:32:43.204427 22215 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.62:35385
I20260925 07:32:43.204452 22283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.62:35385 every 8 connection(s)
I20260925 07:32:43.204814 22215 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
I20260925 07:32:43.206151 22284 sys_catalog.cc:263] Verifying existing consensus state
I20260925 07:32:43.206777 22284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7c638cf5b1545c787782729be7883b5: Bootstrap starting.
I20260925 07:32:43.207669 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 22215
I20260925 07:32:43.207957 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.66.20:54757/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.66.1:46341
--local_ip_for_outbound_sockets=127.9.66.1
--tserver_master_addrs=127.9.66.62:35385
--webserver_port=34123
--webserver_interface=127.9.66.1
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.1.keytab
--principal=kudu/127.9.66.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:46093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.9.66.20:54757/kms with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 07:32:43.216835 22284 log.cc:824] T 00000000000000000000000000000000 P d7c638cf5b1545c787782729be7883b5: Log is configured to *not* fsync() on all Append() calls
I20260925 07:32:43.219250 22284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7c638cf5b1545c787782729be7883b5: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 07:32:43.219578 22284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7c638cf5b1545c787782729be7883b5: Bootstrap complete.
I20260925 07:32:43.222872 22284 raft_consensus.cc:348] T 00000000000000000000000000000000 P d7c638cf5b1545c787782729be7883b5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7c638cf5b1545c787782729be7883b5" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 35385 } }
I20260925 07:32:43.223197 22284 raft_consensus.cc:729] T 00000000000000000000000000000000 P d7c638cf5b1545c787782729be7883b5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7c638cf5b1545c787782729be7883b5, State: Initialized, Role: FOLLOWER
I20260925 07:32:43.223385 22284 consensus_queue.cc:260] T 00000000000000000000000000000000 P d7c638cf5b1545c787782729be7883b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7c638cf5b1545c787782729be7883b5" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 35385 } }
I20260925 07:32:43.223448 22284 raft_consensus.cc:388] T 00000000000000000000000000000000 P d7c638cf5b1545c787782729be7883b5 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 07:32:43.223491 22284 raft_consensus.cc:482] T 00000000000000000000000000000000 P d7c638cf5b1545c787782729be7883b5 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 07:32:43.223528 22284 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d7c638cf5b1545c787782729be7883b5 [term 1 FOLLOWER]: Advancing to term 2
I20260925 07:32:43.224756 22284 raft_consensus.cc:504] T 00000000000000000000000000000000 P d7c638cf5b1545c787782729be7883b5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7c638cf5b1545c787782729be7883b5" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 35385 } }
I20260925 07:32:43.224856 22284 leader_election.cc:302] T 00000000000000000000000000000000 P d7c638cf5b1545c787782729be7883b5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d7c638cf5b1545c787782729be7883b5; no voters:
I20260925 07:32:43.224998 22284 leader_election.cc:288] T 00000000000000000000000000000000 P d7c638cf5b1545c787782729be7883b5 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 07:32:43.225121 22288 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d7c638cf5b1545c787782729be7883b5 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 07:32:43.225225 22284 sys_catalog.cc:564] T 00000000000000000000000000000000 P d7c638cf5b1545c787782729be7883b5 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:32:43.225337 22288 raft_consensus.cc:686] T 00000000000000000000000000000000 P d7c638cf5b1545c787782729be7883b5 [term 2 LEADER]: Becoming Leader. State: Replica: d7c638cf5b1545c787782729be7883b5, State: Running, Role: LEADER
I20260925 07:32:43.225471 22288 consensus_queue.cc:237] T 00000000000000000000000000000000 P d7c638cf5b1545c787782729be7883b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7c638cf5b1545c787782729be7883b5" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 35385 } }
I20260925 07:32:43.225823 22288 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7c638cf5b1545c787782729be7883b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "d7c638cf5b1545c787782729be7883b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7c638cf5b1545c787782729be7883b5" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 35385 } } }
I20260925 07:32:43.225952 22288 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7c638cf5b1545c787782729be7883b5 [sys.catalog]: This master's current role is: LEADER
I20260925 07:32:43.226130 22288 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7c638cf5b1545c787782729be7883b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d7c638cf5b1545c787782729be7883b5. Latest consensus state: current_term: 2 leader_uuid: "d7c638cf5b1545c787782729be7883b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7c638cf5b1545c787782729be7883b5" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 35385 } } }
I20260925 07:32:43.226238 22288 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7c638cf5b1545c787782729be7883b5 [sys.catalog]: This master's current role is: LEADER
I20260925 07:32:43.226363 22297 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 07:32:43.226601 22284 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 07:32:43.226912 22297 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 07:32:43.227105 22297 catalog_manager.cc:1259] Loaded cluster ID: e7d9ed68c344403b903daa170320b951
I20260925 07:32:43.227173 22297 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 07:32:43.231689 22297 catalog_manager.cc:1504] Loading token signing keys...
I20260925 07:32:43.232035 22297 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d7c638cf5b1545c787782729be7883b5: Loaded TSK: 0
I20260925 07:32:43.232476 22297 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 07:32:43.322693 22286 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:32:43.322942 22286 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:32:43.322978 22286 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:32:43.324626 22286 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:32:43.324689 22286 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:32:43.324731 22286 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.66.1
I20260925 07:32:43.326913 22286 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:46093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.9.66.20:54757/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.1.keytab
--principal=kudu/127.9.66.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.66.1:46341
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.66.1
--webserver_port=34123
--tserver_master_addrs=127.9.66.62:35385
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22286
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.66.1
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:32:43.327159 22286 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:32:43.327399 22286 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:32:43.333616 22322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:32:43.333675 22321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:32:43.334028 22324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:32:43.334229 22286 server_base.cc:1034] running on GCE node
I20260925 07:32:43.334435 22286 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:32:43.334698 22286 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:32:43.335879 22286 hybrid_clock.cc:648] HybridClock initialized: now 1790321563335847 us; error 47 us; skew 500 ppm
Sep 25 07:32:43 dist-test-slave-57fn krb5kdc[21252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321563, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:32:43.339821 22286 init.cc:377] Logged in from keytab as kudu/127.9.66.1@KRBTEST.COM (short username kudu)
I20260925 07:32:43.340322 22286 webserver.cc:460] Webserver started at http://127.9.66.1:34123/ using document root <none> and password file <none>
I20260925 07:32:43.340602 22286 fs_manager.cc:359] Metadata directory not provided
I20260925 07:32:43.340687 22286 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 07:32:43 dist-test-slave-57fn krb5kdc[21252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321563, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.1@KRBTEST.COM for HTTP/127.9.66.20@KRBTEST.COM
I20260925 07:32:43.359537 22286 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:32:43.363631 22332 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:32:43.363842 22286 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.000s
I20260925 07:32:43.363925 22286 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/ts-0/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/ts-0/wal
uuid: "528e95197ba948cf9fbb1cdfae5f3def"
format_stamp: "Formatted at 2026-09-25 07:32:42 on dist-test-slave-57fn"
server_key: "294d6fd2de939bcba60e21f9fcc7a1cf"
server_key_iv: "b059387ffb3dee1aa8ddfa148e6c611e"
server_key_version: "kuduclusterkey@0"
I20260925 07:32:43.364364 22286 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:32:43.376965 22286 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:32:43.379304 22286 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:32:43.379460 22286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:32:43.379714 22286 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:32:43.380162 22286 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:32:43.380215 22286 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:32:43.380249 22286 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:32:43.380267 22286 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:32:43.387439 22286 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.1:46341
I20260925 07:32:43.387604 22445 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.1:46341 every 8 connection(s)
I20260925 07:32:43.387895 22286 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/ts-0/data/info.pb
I20260925 07:32:43.390395 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 22286
I20260925 07:32:43.390688 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.66.20:54757/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.66.2:36895
--local_ip_for_outbound_sockets=127.9.66.2
--tserver_master_addrs=127.9.66.62:35385
--webserver_port=43867
--webserver_interface=127.9.66.2
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.2.keytab
--principal=kudu/127.9.66.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:46093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.9.66.20:54757/kms with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 07:32:43 dist-test-slave-57fn krb5kdc[21252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321563, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.1@KRBTEST.COM for kudu/127.9.66.62@KRBTEST.COM
I20260925 07:32:43.401650 22449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:43.388755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.1:34523 (local address 127.9.66.62:35385)
0925 07:32:43.388955 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0925 07:32:43.388963 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 07:32:43.389818 (+ 855us) server_negotiation.cc:407] Connection header received
0925 07:32:43.390590 (+ 772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:32:43.390598 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:32:43.390652 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:32:43.390781 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:32:43.391607 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:32:43.392972 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:32:43.394460 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:32:43.395061 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:32:43.395141 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:32:43.397726 (+ 2585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:32:43.397730 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:32:43.397730 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:32:43.397741 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:32:43.399729 (+ 1988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:32:43.400198 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:32:43.400201 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:32:43.400202 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:32:43.400300 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:32:43.400598 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:32:43.400599 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:32:43.400599 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:32:43.400860 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:32:43.400918 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 07:32:43.401251 (+ 333us) server_negotiation.cc:299] Negotiation successful
0925 07:32:43.401367 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":110,"threads_started":1}
I20260925 07:32:43.402089 22448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:43.388649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:35385 (local address 127.9.66.1:34523)
0925 07:32:43.389628 (+ 979us) negotiation.cc:107] Waiting for socket to connect
0925 07:32:43.389673 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 07:32:43.390326 (+ 653us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:32:43.390979 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:32:43.390983 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:32:43.391211 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:32:43.391464 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:32:43.391469 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:32:43.393100 (+ 1631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:32:43.393101 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:32:43.394355 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:32:43.394358 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:32:43.395283 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:32:43.395284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:32:43.395409 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:32:43.395996 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:32:43.396005 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:32:43.397605 (+ 1600us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:32:43.399824 (+ 2219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:32:43.399830 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:32:43.399832 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:32:43.400102 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:32:43.400371 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:32:43.400371 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:32:43.400372 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:32:43.400487 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:32:43.400951 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:32:43.400956 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:32:43.401088 (+ 132us) client_negotiation.cc:769] Sending connection context
0925 07:32:43.401748 (+ 660us) client_negotiation.cc:240] Negotiation successful
0925 07:32:43.401889 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":938,"thread_start_us":85,"threads_started":1}
I20260925 07:32:43.403007 22446 heartbeater.cc:347] Connected to a master server at 127.9.66.62:35385
I20260925 07:32:43.403103 22446 heartbeater.cc:464] Registering TS with master...
I20260925 07:32:43.403278 22446 heartbeater.cc:511] Master 127.9.66.62:35385 requested a full tablet report, sending...
W20260925 07:32:43.404389 22446 heartbeater.cc:644] Failed to heartbeat to 127.9.66.62:35385 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 07:32:43.404459 22446 heartbeater.cc:464] Registering TS with master...
I20260925 07:32:43.404520 22446 heartbeater.cc:511] Master 127.9.66.62:35385 requested a full tablet report, sending...
I20260925 07:32:43.408449 22446 heartbeater.cc:464] Registering TS with master...
I20260925 07:32:43.408576 22446 heartbeater.cc:511] Master 127.9.66.62:35385 requested a full tablet report, sending...
W20260925 07:32:43.408933 22446 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 07:32:43.505399 22450 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:32:43.505707 22450 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:32:43.505743 22450 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:32:43.507545 22450 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:32:43.507656 22450 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:32:43.507752 22450 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.66.2
I20260925 07:32:43.510075 22450 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:46093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.9.66.20:54757/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.2.keytab
--principal=kudu/127.9.66.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.66.2:36895
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.66.2
--webserver_port=43867
--tserver_master_addrs=127.9.66.62:35385
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22450
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.66.2
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:32:43.510324 22450 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:32:43.510582 22450 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:32:43.516892 22458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:32:43.516882 22456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:32:43.517356 22455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:32:43.518416 22450 server_base.cc:1034] running on GCE node
I20260925 07:32:43.518612 22450 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:32:43.518903 22450 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:32:43.520088 22450 hybrid_clock.cc:648] HybridClock initialized: now 1790321563520073 us; error 28 us; skew 500 ppm
Sep 25 07:32:43 dist-test-slave-57fn krb5kdc[21252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321563, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:32:43.522862 22450 init.cc:377] Logged in from keytab as kudu/127.9.66.2@KRBTEST.COM (short username kudu)
I20260925 07:32:43.523375 22450 webserver.cc:460] Webserver started at http://127.9.66.2:43867/ using document root <none> and password file <none>
I20260925 07:32:43.523630 22450 fs_manager.cc:359] Metadata directory not provided
I20260925 07:32:43.523697 22450 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 07:32:43 dist-test-slave-57fn krb5kdc[21252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321563, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.2@KRBTEST.COM for HTTP/127.9.66.20@KRBTEST.COM
I20260925 07:32:43.540865 22450 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:32:43.541771 22466 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:32:43.541939 22450 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 07:32:43.542033 22450 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/ts-1/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/ts-1/wal
uuid: "02d6b2ab7add458193cf60d80f814b44"
format_stamp: "Formatted at 2026-09-25 07:32:42 on dist-test-slave-57fn"
server_key: "fc500c1e801e01978d2e7dd43cf77c31"
server_key_iv: "2e2d3735e8b9ee5fb9881d2c74d2a104"
server_key_version: "kuduclusterkey@0"
I20260925 07:32:43.542909 22450 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:32:43.566779 22450 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:32:43.569283 22450 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:32:43.569449 22450 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:32:43.569761 22450 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:32:43.570163 22450 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:32:43.570230 22450 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:32:43.570271 22450 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:32:43.570297 22450 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:32:43.578200 22450 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.2:36895
I20260925 07:32:43.578336 22579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.2:36895 every 8 connection(s)
I20260925 07:32:43.578640 22450 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/ts-1/data/info.pb
Sep 25 07:32:43 dist-test-slave-57fn krb5kdc[21252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321563, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.2@KRBTEST.COM for kudu/127.9.66.62@KRBTEST.COM
I20260925 07:32:43.588063 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 22450
I20260925 07:32:43.588367 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.66.20:54757/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.66.3:33415
--local_ip_for_outbound_sockets=127.9.66.3
--tserver_master_addrs=127.9.66.62:35385
--webserver_port=44015
--webserver_interface=127.9.66.3
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.3.keytab
--principal=kudu/127.9.66.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:46093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.9.66.20:54757/kms with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 07:32:43.590802 22449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:43.579418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.2:35217 (local address 127.9.66.62:35385)
0925 07:32:43.579684 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0925 07:32:43.579687 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:32:43.579728 (+ 41us) server_negotiation.cc:407] Connection header received
0925 07:32:43.580248 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:32:43.580250 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:32:43.580265 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:32:43.580318 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:32:43.580875 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:32:43.581391 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:32:43.582361 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:32:43.582660 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:32:43.582751 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:32:43.585777 (+ 3026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:32:43.585778 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:32:43.585779 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:32:43.585786 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:32:43.587535 (+ 1749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:32:43.589804 (+ 2269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:32:43.589807 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:32:43.589807 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:32:43.589865 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:32:43.590139 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:32:43.590140 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:32:43.590140 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:32:43.590264 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:32:43.590340 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 07:32:43.590621 (+ 281us) server_negotiation.cc:299] Negotiation successful
0925 07:32:43.590714 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241}
I20260925 07:32:43.590979 22582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:43.579436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:35385 (local address 127.9.66.2:35217)
0925 07:32:43.579610 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 07:32:43.579652 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 07:32:43.580140 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:32:43.580395 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:32:43.580399 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:32:43.580590 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:32:43.580776 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:32:43.580781 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:32:43.581483 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:32:43.581484 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:32:43.582266 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:32:43.582268 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:32:43.582775 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:32:43.582776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:32:43.582888 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:32:43.583507 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:32:43.583527 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:32:43.585670 (+ 2143us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:32:43.587628 (+ 1958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:32:43.587634 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:32:43.587636 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:32:43.589710 (+ 2074us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:32:43.589931 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:32:43.589932 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:32:43.589932 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:32:43.590059 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:32:43.590333 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:32:43.590339 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:32:43.590484 (+ 145us) client_negotiation.cc:769] Sending connection context
0925 07:32:43.590590 (+ 106us) client_negotiation.cc:240] Negotiation successful
0925 07:32:43.590741 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":107,"threads_started":1}
I20260925 07:32:43.591475 22580 heartbeater.cc:347] Connected to a master server at 127.9.66.62:35385
I20260925 07:32:43.591578 22580 heartbeater.cc:464] Registering TS with master...
I20260925 07:32:43.591776 22580 heartbeater.cc:511] Master 127.9.66.62:35385 requested a full tablet report, sending...
W20260925 07:32:43.592505 22580 heartbeater.cc:644] Failed to heartbeat to 127.9.66.62:35385 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 07:32:43.592584 22580 heartbeater.cc:464] Registering TS with master...
I20260925 07:32:43.592657 22580 heartbeater.cc:511] Master 127.9.66.62:35385 requested a full tablet report, sending...
I20260925 07:32:43.593060 22580 heartbeater.cc:464] Registering TS with master...
I20260925 07:32:43.593145 22580 heartbeater.cc:511] Master 127.9.66.62:35385 requested a full tablet report, sending...
W20260925 07:32:43.593485 22580 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 07:32:43.688269 22583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:32:43.688563 22583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:32:43.688604 22583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:32:43.689992 22583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:32:43.690047 22583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:32:43.690097 22583 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.66.3
I20260925 07:32:43.691671 22583 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:46093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.9.66.20:54757/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.3.keytab
--principal=kudu/127.9.66.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.66.3:33415
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.66.3
--webserver_port=44015
--tserver_master_addrs=127.9.66.62:35385
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22583
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.66.3
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:32:43.691911 22583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:32:43.692142 22583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:32:43.697134 22589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:32:43.697204 22588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:32:43.697134 22591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:32:43.697616 22583 server_base.cc:1034] running on GCE node
I20260925 07:32:43.697808 22583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:32:43.698062 22583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:32:43.699213 22583 hybrid_clock.cc:648] HybridClock initialized: now 1790321563699201 us; error 35 us; skew 500 ppm
Sep 25 07:32:43 dist-test-slave-57fn krb5kdc[21252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321563, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:32:43.702306 22583 init.cc:377] Logged in from keytab as kudu/127.9.66.3@KRBTEST.COM (short username kudu)
I20260925 07:32:43.702834 22583 webserver.cc:460] Webserver started at http://127.9.66.3:44015/ using document root <none> and password file <none>
I20260925 07:32:43.703102 22583 fs_manager.cc:359] Metadata directory not provided
I20260925 07:32:43.703179 22583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 07:32:43 dist-test-slave-57fn krb5kdc[21252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321563, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.3@KRBTEST.COM for HTTP/127.9.66.20@KRBTEST.COM
I20260925 07:32:43.722594 22583 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.001s
I20260925 07:32:43.723752 22600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:32:43.723919 22583 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 07:32:43.724025 22583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/ts-2/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/ts-2/wal
uuid: "fa14e49f1dd243b6aa588d65fabe6c9a"
format_stamp: "Formatted at 2026-09-25 07:32:42 on dist-test-slave-57fn"
server_key: "55b945c1674a4707a053df64f5305780"
server_key_iv: "2f3cd8e30f1b39d263bd0d8c280682ff"
server_key_version: "kuduclusterkey@0"
I20260925 07:32:43.724483 22583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:32:43.752569 22583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:32:43.754853 22583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:32:43.754994 22583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:32:43.755213 22583 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:32:43.755532 22583 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:32:43.755570 22583 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:32:43.755599 22583 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:32:43.755621 22583 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:32:43.762219 22583 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.3:33415
I20260925 07:32:43.762413 22713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.3:33415 every 8 connection(s)
I20260925 07:32:43.762650 22583 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/ts-2/data/info.pb
I20260925 07:32:43.764940 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 22583
Sep 25 07:32:43 dist-test-slave-57fn krb5kdc[21252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321563, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.3@KRBTEST.COM for kudu/127.9.66.62@KRBTEST.COM
I20260925 07:32:43.773806 22717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:43.765372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59898 (local address 127.9.66.62:35385)
0925 07:32:43.765569 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 07:32:43.765571 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:32:43.765610 (+ 39us) server_negotiation.cc:407] Connection header received
0925 07:32:43.765673 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:32:43.765674 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:32:43.765683 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:32:43.765750 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:32:43.766477 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:32:43.767654 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:32:43.768859 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:32:43.769183 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:32:43.769266 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:32:43.770755 (+ 1489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:32:43.770756 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:32:43.770757 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:32:43.770764 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:32:43.772491 (+ 1727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:32:43.772908 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:32:43.772909 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:32:43.772909 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:32:43.772959 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:32:43.773168 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:32:43.773169 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:32:43.773170 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:32:43.773320 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:32:43.773569 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0925 07:32:43.773613 (+ 44us) server_negotiation.cc:299] Negotiation successful
0925 07:32:43.773706 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":60,"threads_started":1}
I20260925 07:32:43.774178 22449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:43.763395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.3:37095 (local address 127.9.66.62:35385)
0925 07:32:43.763454 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 07:32:43.763456 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:32:43.763659 (+ 203us) server_negotiation.cc:407] Connection header received
0925 07:32:43.764177 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:32:43.764179 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:32:43.764193 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:32:43.764238 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:32:43.765193 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:32:43.765915 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:32:43.767356 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:32:43.767629 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:32:43.767692 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:32:43.770218 (+ 2526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:32:43.770220 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:32:43.770220 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:32:43.770227 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:32:43.772165 (+ 1938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:32:43.772652 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:32:43.772653 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:32:43.772654 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:32:43.772721 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:32:43.772993 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:32:43.772994 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:32:43.772994 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:32:43.773140 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:32:43.773201 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 07:32:43.773800 (+ 599us) server_negotiation.cc:299] Negotiation successful
0925 07:32:43.774087 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260925 07:32:43.774745 22716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:43.763369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:35385 (local address 127.9.66.3:37095)
0925 07:32:43.763529 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 07:32:43.763563 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 07:32:43.764063 (+ 500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:32:43.764315 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:32:43.764319 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:32:43.764492 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:32:43.764703 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:32:43.764707 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:32:43.766034 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:32:43.766035 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:32:43.767257 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:32:43.767260 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:32:43.767776 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:32:43.767777 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:32:43.767891 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:32:43.768499 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:32:43.768507 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:32:43.770084 (+ 1577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:32:43.772255 (+ 2171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:32:43.772262 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:32:43.772264 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:32:43.772567 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:32:43.772789 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:32:43.772790 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:32:43.772790 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:32:43.772924 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:32:43.773192 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:32:43.773197 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:32:43.773343 (+ 146us) client_negotiation.cc:769] Sending connection context
0925 07:32:43.773587 (+ 244us) client_negotiation.cc:240] Negotiation successful
0925 07:32:43.774474 (+ 887us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":70,"threads_started":1}
I20260925 07:32:43.775187 22714 heartbeater.cc:347] Connected to a master server at 127.9.66.62:35385
I20260925 07:32:43.775300 22714 heartbeater.cc:464] Registering TS with master...
I20260925 07:32:43.775476 22714 heartbeater.cc:511] Master 127.9.66.62:35385 requested a full tablet report, sending...
W20260925 07:32:43.776087 22714 heartbeater.cc:644] Failed to heartbeat to 127.9.66.62:35385 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 07:32:43.776163 22714 heartbeater.cc:464] Registering TS with master...
I20260925 07:32:43.776232 22714 heartbeater.cc:511] Master 127.9.66.62:35385 requested a full tablet report, sending...
I20260925 07:32:43.776520 22714 heartbeater.cc:464] Registering TS with master...
I20260925 07:32:43.776602 22714 heartbeater.cc:511] Master 127.9.66.62:35385 requested a full tablet report, sending...
W20260925 07:32:43.776891 22714 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 25 07:32:44 dist-test-slave-57fn krb5kdc[21252](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790321564, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:32:44.409148 22446 heartbeater.cc:464] Registering TS with master...
I20260925 07:32:44.409302 22446 heartbeater.cc:511] Master 127.9.66.62:35385 requested a full tablet report, sending...
I20260925 07:32:44.409703 22248 ts_manager.cc:194] Registered new tserver with Master: 528e95197ba948cf9fbb1cdfae5f3def (127.9.66.1:46341)
I20260925 07:32:44.410635 22248 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.66.1@KRBTEST.COM'} at 127.9.66.1:34523
I20260925 07:32:44.593696 22580 heartbeater.cc:464] Registering TS with master...
I20260925 07:32:44.593851 22580 heartbeater.cc:511] Master 127.9.66.62:35385 requested a full tablet report, sending...
I20260925 07:32:44.594202 22248 ts_manager.cc:194] Registered new tserver with Master: 02d6b2ab7add458193cf60d80f814b44 (127.9.66.2:36895)
I20260925 07:32:44.594736 22248 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.66.2@KRBTEST.COM'} at 127.9.66.2:35217
Sep 25 07:32:44 dist-test-slave-57fn krb5kdc[21252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321564, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.62@KRBTEST.COM for HTTP/127.9.66.20@KRBTEST.COM
I20260925 07:32:44.777052 22714 heartbeater.cc:464] Registering TS with master...
I20260925 07:32:44.777179 22714 heartbeater.cc:511] Master 127.9.66.62:35385 requested a full tablet report, sending...
I20260925 07:32:44.777472 22248 ts_manager.cc:194] Registered new tserver with Master: fa14e49f1dd243b6aa588d65fabe6c9a (127.9.66.3:33415)
I20260925 07:32:44.778046 22248 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.66.3@KRBTEST.COM'} at 127.9.66.3:37095
I20260925 07:32:44.787160 9480 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 07:32:44.799522 22736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:44.788095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59900 (local address 127.9.66.62:35385)
0925 07:32:44.788206 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0925 07:32:44.788208 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:32:44.788267 (+ 59us) server_negotiation.cc:407] Connection header received
0925 07:32:44.788333 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:32:44.788334 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:32:44.788346 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:32:44.788401 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:32:44.788951 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:32:44.789448 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:32:44.790174 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:32:44.790384 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:32:44.790457 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:32:44.796834 (+ 6377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:32:44.796835 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:32:44.796836 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:32:44.796848 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:32:44.798358 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:32:44.798735 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:32:44.798736 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:32:44.798737 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:32:44.798796 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:32:44.798997 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:32:44.798998 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:32:44.798998 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:32:44.799108 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:32:44.799206 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 07:32:44.799311 (+ 105us) server_negotiation.cc:299] Negotiation successful
0925 07:32:44.799392 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88,"thread_start_us":57,"threads_started":1}
I20260925 07:32:44.801332 22248 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59900:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 07:32:44.803164 22248 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 07:32:44.811813 22741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:44.806073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.1:46341 (local address 127.0.0.1:50450)
0925 07:32:44.807452 (+ 1379us) negotiation.cc:107] Waiting for socket to connect
0925 07:32:44.807461 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 07:32:44.807507 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:32:44.807864 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:32:44.807864 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:32:44.807867 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:32:44.807932 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:32:44.807933 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:32:44.809210 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:32:44.809210 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:32:44.810590 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:32:44.810593 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:32:44.811543 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:32:44.811544 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:32:44.811620 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:32:44.811632 (+ 12us) client_negotiation.cc:769] Sending connection context
0925 07:32:44.811692 (+ 60us) client_negotiation.cc:240] Negotiation successful
0925 07:32:44.811738 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1366,"mutex_wait_us":309,"thread_start_us":81,"threads_started":1}
I20260925 07:32:44.812117 22743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:44.806399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50450 (local address 127.9.66.1:46341)
0925 07:32:44.807072 (+ 673us) server_negotiation.cc:206] Beginning negotiation
0925 07:32:44.807077 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 07:32:44.807584 (+ 507us) server_negotiation.cc:407] Connection header received
0925 07:32:44.807625 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:32:44.807628 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:32:44.807646 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:32:44.807751 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:32:44.808015 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:32:44.809110 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:32:44.810687 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:32:44.811383 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:32:44.811454 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:32:44.811510 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 07:32:44.812001 (+ 491us) server_negotiation.cc:299] Negotiation successful
0925 07:32:44.812044 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":639,"thread_start_us":55,"threads_started":1}
I20260925 07:32:44.812471 22745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:44.806057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.3:33415 (local address 127.0.0.1:36252)
0925 07:32:44.807231 (+ 1174us) negotiation.cc:107] Waiting for socket to connect
0925 07:32:44.807270 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 07:32:44.807346 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:32:44.807613 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:32:44.807616 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:32:44.807619 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:32:44.807679 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:32:44.807681 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:32:44.809220 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:32:44.809221 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:32:44.810608 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:32:44.810611 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:32:44.812111 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:32:44.812112 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:32:44.812309 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:32:44.812313 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 07:32:44.812361 (+ 48us) client_negotiation.cc:240] Negotiation successful
0925 07:32:44.812396 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1147,"mutex_wait_us":46,"thread_start_us":75,"threads_started":1}
I20260925 07:32:44.812770 22380 tablet_service.cc:1467] Processing CreateTablet for tablet 027f81f3fa784fec965261678c71a01a (DEFAULT_TABLE table=test-table [id=ce9dca80d988435db8ad619817b027d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:32:44.812899 22744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:44.806536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36252 (local address 127.9.66.3:33415)
0925 07:32:44.807166 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0925 07:32:44.807170 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 07:32:44.807353 (+ 183us) server_negotiation.cc:407] Connection header received
0925 07:32:44.807422 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:32:44.807424 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:32:44.807441 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:32:44.807521 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:32:44.807763 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:32:44.809122 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:32:44.810709 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:32:44.811766 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:32:44.811835 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:32:44.812761 (+ 926us) server_negotiation.cc:1035] Waiting for connection context
0925 07:32:44.812789 (+ 28us) server_negotiation.cc:299] Negotiation successful
0925 07:32:44.812828 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":600,"thread_start_us":54,"threads_started":1}
I20260925 07:32:44.813118 22380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 027f81f3fa784fec965261678c71a01a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:32:44.814399 22648 tablet_service.cc:1467] Processing CreateTablet for tablet 027f81f3fa784fec965261678c71a01a (DEFAULT_TABLE table=test-table [id=ce9dca80d988435db8ad619817b027d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:32:44.814700 22648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 027f81f3fa784fec965261678c71a01a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:32:44.816419 22747 tablet_bootstrap.cc:492] T 027f81f3fa784fec965261678c71a01a P 528e95197ba948cf9fbb1cdfae5f3def: Bootstrap starting.
I20260925 07:32:44.817006 22747 tablet_bootstrap.cc:654] T 027f81f3fa784fec965261678c71a01a P 528e95197ba948cf9fbb1cdfae5f3def: Neither blocks nor log segments found. Creating new log.
I20260925 07:32:44.817282 22747 log.cc:824] T 027f81f3fa784fec965261678c71a01a P 528e95197ba948cf9fbb1cdfae5f3def: Log is configured to *not* fsync() on all Append() calls
I20260925 07:32:44.817596 22748 tablet_bootstrap.cc:492] T 027f81f3fa784fec965261678c71a01a P fa14e49f1dd243b6aa588d65fabe6c9a: Bootstrap starting.
I20260925 07:32:44.818210 22748 tablet_bootstrap.cc:654] T 027f81f3fa784fec965261678c71a01a P fa14e49f1dd243b6aa588d65fabe6c9a: Neither blocks nor log segments found. Creating new log.
I20260925 07:32:44.818490 22748 log.cc:824] T 027f81f3fa784fec965261678c71a01a P fa14e49f1dd243b6aa588d65fabe6c9a: Log is configured to *not* fsync() on all Append() calls
I20260925 07:32:44.819213 22747 tablet_bootstrap.cc:492] T 027f81f3fa784fec965261678c71a01a P 528e95197ba948cf9fbb1cdfae5f3def: No bootstrap required, opened a new log
I20260925 07:32:44.819279 22747 ts_tablet_manager.cc:1397] T 027f81f3fa784fec965261678c71a01a P 528e95197ba948cf9fbb1cdfae5f3def: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260925 07:32:44.821682 22747 raft_consensus.cc:348] T 027f81f3fa784fec965261678c71a01a P 528e95197ba948cf9fbb1cdfae5f3def [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "528e95197ba948cf9fbb1cdfae5f3def" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 46341 } } peers { permanent_uuid: "fa14e49f1dd243b6aa588d65fabe6c9a" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 33415 } } peers { permanent_uuid: "02d6b2ab7add458193cf60d80f814b44" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 36895 } }
I20260925 07:32:44.821843 22747 raft_consensus.cc:374] T 027f81f3fa784fec965261678c71a01a P 528e95197ba948cf9fbb1cdfae5f3def [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:32:44.821877 22747 raft_consensus.cc:729] T 027f81f3fa784fec965261678c71a01a P 528e95197ba948cf9fbb1cdfae5f3def [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 528e95197ba948cf9fbb1cdfae5f3def, State: Initialized, Role: FOLLOWER
I20260925 07:32:44.822006 22747 consensus_queue.cc:260] T 027f81f3fa784fec965261678c71a01a P 528e95197ba948cf9fbb1cdfae5f3def [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "528e95197ba948cf9fbb1cdfae5f3def" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 46341 } } peers { permanent_uuid: "fa14e49f1dd243b6aa588d65fabe6c9a" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 33415 } } peers { permanent_uuid: "02d6b2ab7add458193cf60d80f814b44" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 36895 } }
I20260925 07:32:44.822230 22747 ts_tablet_manager.cc:1428] T 027f81f3fa784fec965261678c71a01a P 528e95197ba948cf9fbb1cdfae5f3def: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 07:32:44.822343 22446 heartbeater.cc:503] Master 127.9.66.62:35385 was elected leader, sending a full tablet report...
I20260925 07:32:44.823554 22748 tablet_bootstrap.cc:492] T 027f81f3fa784fec965261678c71a01a P fa14e49f1dd243b6aa588d65fabe6c9a: No bootstrap required, opened a new log
I20260925 07:32:44.823621 22748 ts_tablet_manager.cc:1397] T 027f81f3fa784fec965261678c71a01a P fa14e49f1dd243b6aa588d65fabe6c9a: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260925 07:32:44.824527 22742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:44.806738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.2:36895 (local address 127.0.0.1:42072)
0925 07:32:44.814103 (+ 7365us) negotiation.cc:107] Waiting for socket to connect
0925 07:32:44.814116 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 07:32:44.814164 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:32:44.817871 (+ 3707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:32:44.817873 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:32:44.817876 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:32:44.817921 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:32:44.817922 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:32:44.821712 (+ 3790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:32:44.821714 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:32:44.823194 (+ 1480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:32:44.823197 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:32:44.824305 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:32:44.824306 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:32:44.824363 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:32:44.824367 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 07:32:44.824419 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 07:32:44.824458 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2712,"thread_start_us":64,"threads_started":1}
I20260925 07:32:44.825048 22746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:44.806853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42072 (local address 127.9.66.2:36895)
0925 07:32:44.817608 (+ 10755us) server_negotiation.cc:206] Beginning negotiation
0925 07:32:44.817612 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 07:32:44.817624 (+ 12us) server_negotiation.cc:407] Connection header received
0925 07:32:44.817659 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:32:44.817661 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:32:44.817676 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:32:44.817764 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:32:44.818026 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:32:44.819239 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:32:44.823298 (+ 4059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:32:44.824152 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:32:44.824220 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:32:44.824271 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 07:32:44.824923 (+ 652us) server_negotiation.cc:299] Negotiation successful
0925 07:32:44.824964 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10718,"thread_start_us":59,"threads_started":1}
I20260925 07:32:44.825754 22514 tablet_service.cc:1467] Processing CreateTablet for tablet 027f81f3fa784fec965261678c71a01a (DEFAULT_TABLE table=test-table [id=ce9dca80d988435db8ad619817b027d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:32:44.826054 22514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 027f81f3fa784fec965261678c71a01a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:32:44.829291 22752 tablet_bootstrap.cc:492] T 027f81f3fa784fec965261678c71a01a P 02d6b2ab7add458193cf60d80f814b44: Bootstrap starting.
I20260925 07:32:44.830055 22752 tablet_bootstrap.cc:654] T 027f81f3fa784fec965261678c71a01a P 02d6b2ab7add458193cf60d80f814b44: Neither blocks nor log segments found. Creating new log.
I20260925 07:32:44.830118 22748 raft_consensus.cc:348] T 027f81f3fa784fec965261678c71a01a P fa14e49f1dd243b6aa588d65fabe6c9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "528e95197ba948cf9fbb1cdfae5f3def" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 46341 } } peers { permanent_uuid: "fa14e49f1dd243b6aa588d65fabe6c9a" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 33415 } } peers { permanent_uuid: "02d6b2ab7add458193cf60d80f814b44" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 36895 } }
I20260925 07:32:44.830272 22748 raft_consensus.cc:374] T 027f81f3fa784fec965261678c71a01a P fa14e49f1dd243b6aa588d65fabe6c9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:32:44.830308 22748 raft_consensus.cc:729] T 027f81f3fa784fec965261678c71a01a P fa14e49f1dd243b6aa588d65fabe6c9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa14e49f1dd243b6aa588d65fabe6c9a, State: Initialized, Role: FOLLOWER
I20260925 07:32:44.830432 22748 consensus_queue.cc:260] T 027f81f3fa784fec965261678c71a01a P fa14e49f1dd243b6aa588d65fabe6c9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "528e95197ba948cf9fbb1cdfae5f3def" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 46341 } } peers { permanent_uuid: "fa14e49f1dd243b6aa588d65fabe6c9a" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 33415 } } peers { permanent_uuid: "02d6b2ab7add458193cf60d80f814b44" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 36895 } }
I20260925 07:32:44.830574 22752 log.cc:824] T 027f81f3fa784fec965261678c71a01a P 02d6b2ab7add458193cf60d80f814b44: Log is configured to *not* fsync() on all Append() calls
I20260925 07:32:44.830660 22748 ts_tablet_manager.cc:1428] T 027f81f3fa784fec965261678c71a01a P fa14e49f1dd243b6aa588d65fabe6c9a: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20260925 07:32:44.830773 22714 heartbeater.cc:503] Master 127.9.66.62:35385 was elected leader, sending a full tablet report...
I20260925 07:32:44.831650 22752 tablet_bootstrap.cc:492] T 027f81f3fa784fec965261678c71a01a P 02d6b2ab7add458193cf60d80f814b44: No bootstrap required, opened a new log
I20260925 07:32:44.831712 22752 ts_tablet_manager.cc:1397] T 027f81f3fa784fec965261678c71a01a P 02d6b2ab7add458193cf60d80f814b44: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 07:32:44.833959 22752 raft_consensus.cc:348] T 027f81f3fa784fec965261678c71a01a P 02d6b2ab7add458193cf60d80f814b44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "528e95197ba948cf9fbb1cdfae5f3def" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 46341 } } peers { permanent_uuid: "fa14e49f1dd243b6aa588d65fabe6c9a" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 33415 } } peers { permanent_uuid: "02d6b2ab7add458193cf60d80f814b44" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 36895 } }
I20260925 07:32:44.834096 22752 raft_consensus.cc:374] T 027f81f3fa784fec965261678c71a01a P 02d6b2ab7add458193cf60d80f814b44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:32:44.834125 22752 raft_consensus.cc:729] T 027f81f3fa784fec965261678c71a01a P 02d6b2ab7add458193cf60d80f814b44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02d6b2ab7add458193cf60d80f814b44, State: Initialized, Role: FOLLOWER
I20260925 07:32:44.834239 22752 consensus_queue.cc:260] T 027f81f3fa784fec965261678c71a01a P 02d6b2ab7add458193cf60d80f814b44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "528e95197ba948cf9fbb1cdfae5f3def" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 46341 } } peers { permanent_uuid: "fa14e49f1dd243b6aa588d65fabe6c9a" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 33415 } } peers { permanent_uuid: "02d6b2ab7add458193cf60d80f814b44" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 36895 } }
I20260925 07:32:44.834452 22752 ts_tablet_manager.cc:1428] T 027f81f3fa784fec965261678c71a01a P 02d6b2ab7add458193cf60d80f814b44: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 07:32:44.834483 22580 heartbeater.cc:503] Master 127.9.66.62:35385 was elected leader, sending a full tablet report...
W20260925 07:32:44.889444 22447 tablet.cc:2369] T 027f81f3fa784fec965261678c71a01a P 528e95197ba948cf9fbb1cdfae5f3def: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 25 07:32:44 dist-test-slave-57fn krb5kdc[21252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321564, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.62@KRBTEST.COM for HTTP/127.9.66.20@KRBTEST.COM
I20260925 07:32:44.968991 22755 raft_consensus.cc:482] T 027f81f3fa784fec965261678c71a01a P 02d6b2ab7add458193cf60d80f814b44 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 07:32:44.969179 22755 raft_consensus.cc:504] T 027f81f3fa784fec965261678c71a01a P 02d6b2ab7add458193cf60d80f814b44 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "528e95197ba948cf9fbb1cdfae5f3def" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 46341 } } peers { permanent_uuid: "fa14e49f1dd243b6aa588d65fabe6c9a" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 33415 } } peers { permanent_uuid: "02d6b2ab7add458193cf60d80f814b44" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 36895 } }
I20260925 07:32:44.969759 22755 leader_election.cc:288] T 027f81f3fa784fec965261678c71a01a P 02d6b2ab7add458193cf60d80f814b44 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 528e95197ba948cf9fbb1cdfae5f3def (127.9.66.1:46341), fa14e49f1dd243b6aa588d65fabe6c9a (127.9.66.3:33415)
I20260925 07:32:44.974308 22757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:44.969885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.1:46341 (local address 127.9.66.2:54567)
0925 07:32:44.970540 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0925 07:32:44.970551 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 07:32:44.970635 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:32:44.970821 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:32:44.970822 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:32:44.970824 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:32:44.970902 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:32:44.970904 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:32:44.971840 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:32:44.971841 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:32:44.973102 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:32:44.973105 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:32:44.974009 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:32:44.974011 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:32:44.974114 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:32:44.974119 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 07:32:44.974170 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 07:32:44.974208 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"thread_start_us":117,"threads_started":1}
I20260925 07:32:44.974740 22743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:44.970392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.2:54567 (local address 127.9.66.1:46341)
0925 07:32:44.970446 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 07:32:44.970448 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:32:44.970605 (+ 157us) server_negotiation.cc:407] Connection header received
0925 07:32:44.970713 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:32:44.970714 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:32:44.970727 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:32:44.970756 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:32:44.970975 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:32:44.971738 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:32:44.973191 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:32:44.973824 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:32:44.973923 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:32:44.973935 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0925 07:32:44.974635 (+ 700us) server_negotiation.cc:299] Negotiation successful
0925 07:32:44.974677 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 07:32:44.975340 22400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "027f81f3fa784fec965261678c71a01a" candidate_uuid: "02d6b2ab7add458193cf60d80f814b44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "528e95197ba948cf9fbb1cdfae5f3def" is_pre_election: true
I20260925 07:32:44.975499 22400 raft_consensus.cc:2454] T 027f81f3fa784fec965261678c71a01a P 528e95197ba948cf9fbb1cdfae5f3def [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02d6b2ab7add458193cf60d80f814b44 in term 0.
I20260925 07:32:44.975732 22470 leader_election.cc:302] T 027f81f3fa784fec965261678c71a01a P 02d6b2ab7add458193cf60d80f814b44 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02d6b2ab7add458193cf60d80f814b44, 528e95197ba948cf9fbb1cdfae5f3def; no voters:
I20260925 07:32:44.975852 22755 raft_consensus.cc:2781] T 027f81f3fa784fec965261678c71a01a P 02d6b2ab7add458193cf60d80f814b44 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 07:32:44.975888 22755 raft_consensus.cc:482] T 027f81f3fa784fec965261678c71a01a P 02d6b2ab7add458193cf60d80f814b44 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 07:32:44.975908 22755 raft_consensus.cc:3037] T 027f81f3fa784fec965261678c71a01a P 02d6b2ab7add458193cf60d80f814b44 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:32:44.976918 22755 raft_consensus.cc:504] T 027f81f3fa784fec965261678c71a01a P 02d6b2ab7add458193cf60d80f814b44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "528e95197ba948cf9fbb1cdfae5f3def" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 46341 } } peers { permanent_uuid: "fa14e49f1dd243b6aa588d65fabe6c9a" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 33415 } } peers { permanent_uuid: "02d6b2ab7add458193cf60d80f814b44" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 36895 } }
I20260925 07:32:44.977038 22755 leader_election.cc:288] T 027f81f3fa784fec965261678c71a01a P 02d6b2ab7add458193cf60d80f814b44 [CANDIDATE]: Term 1 election: Requested vote from peers 528e95197ba948cf9fbb1cdfae5f3def (127.9.66.1:46341), fa14e49f1dd243b6aa588d65fabe6c9a (127.9.66.3:33415)
I20260925 07:32:44.977413 22400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "027f81f3fa784fec965261678c71a01a" candidate_uuid: "02d6b2ab7add458193cf60d80f814b44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "528e95197ba948cf9fbb1cdfae5f3def"
I20260925 07:32:44.977506 22400 raft_consensus.cc:3037] T 027f81f3fa784fec965261678c71a01a P 528e95197ba948cf9fbb1cdfae5f3def [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:32:44.977876 22757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:44.970137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.3:33415 (local address 127.9.66.2:36589)
0925 07:32:44.974375 (+ 4238us) negotiation.cc:107] Waiting for socket to connect
0925 07:32:44.974386 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 07:32:44.974436 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:32:44.974827 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:32:44.974828 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:32:44.974830 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:32:44.974859 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:32:44.974860 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:32:44.975786 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:32:44.975788 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:32:44.976850 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:32:44.976853 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:32:44.977665 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:32:44.977666 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:32:44.977726 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:32:44.977729 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 07:32:44.977777 (+ 48us) client_negotiation.cc:240] Negotiation successful
0925 07:32:44.977811 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4228,"thread_start_us":72,"threads_started":1}
I20260925 07:32:44.978081 22744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:44.970333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.2:36589 (local address 127.9.66.3:33415)
0925 07:32:44.970381 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 07:32:44.970384 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:32:44.974430 (+ 4046us) server_negotiation.cc:407] Connection header received
0925 07:32:44.974516 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:32:44.974518 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:32:44.974530 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:32:44.974573 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:32:44.974972 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:32:44.975690 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:32:44.976962 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:32:44.977554 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:32:44.977622 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:32:44.977633 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 07:32:44.977976 (+ 343us) server_negotiation.cc:299] Negotiation successful
0925 07:32:44.978017 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 07:32:44.978410 22400 raft_consensus.cc:2454] T 027f81f3fa784fec965261678c71a01a P 528e95197ba948cf9fbb1cdfae5f3def [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02d6b2ab7add458193cf60d80f814b44 in term 1.
I20260925 07:32:44.978816 22667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "027f81f3fa784fec965261678c71a01a" candidate_uuid: "02d6b2ab7add458193cf60d80f814b44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa14e49f1dd243b6aa588d65fabe6c9a"
I20260925 07:32:44.978817 22668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "027f81f3fa784fec965261678c71a01a" candidate_uuid: "02d6b2ab7add458193cf60d80f814b44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa14e49f1dd243b6aa588d65fabe6c9a" is_pre_election: true
I20260925 07:32:44.978941 22667 raft_consensus.cc:3037] T 027f81f3fa784fec965261678c71a01a P fa14e49f1dd243b6aa588d65fabe6c9a [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:32:44.979768 22667 raft_consensus.cc:2454] T 027f81f3fa784fec965261678c71a01a P fa14e49f1dd243b6aa588d65fabe6c9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02d6b2ab7add458193cf60d80f814b44 in term 1.
I20260925 07:32:44.980050 22467 leader_election.cc:302] T 027f81f3fa784fec965261678c71a01a P 02d6b2ab7add458193cf60d80f814b44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02d6b2ab7add458193cf60d80f814b44, fa14e49f1dd243b6aa588d65fabe6c9a; no voters:
I20260925 07:32:44.980530 22755 raft_consensus.cc:2781] T 027f81f3fa784fec965261678c71a01a P 02d6b2ab7add458193cf60d80f814b44 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:32:44.980659 22755 raft_consensus.cc:686] T 027f81f3fa784fec965261678c71a01a P 02d6b2ab7add458193cf60d80f814b44 [term 1 LEADER]: Becoming Leader. State: Replica: 02d6b2ab7add458193cf60d80f814b44, State: Running, Role: LEADER
I20260925 07:32:44.980733 22755 consensus_queue.cc:237] T 027f81f3fa784fec965261678c71a01a P 02d6b2ab7add458193cf60d80f814b44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "528e95197ba948cf9fbb1cdfae5f3def" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 46341 } } peers { permanent_uuid: "fa14e49f1dd243b6aa588d65fabe6c9a" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 33415 } } peers { permanent_uuid: "02d6b2ab7add458193cf60d80f814b44" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 36895 } }
I20260925 07:32:44.981572 22248 catalog_manager.cc:5495] T 027f81f3fa784fec965261678c71a01a P 02d6b2ab7add458193cf60d80f814b44 reported cstate change: term changed from 0 to 1, leader changed from <none> to 02d6b2ab7add458193cf60d80f814b44 (127.9.66.2). New cstate: current_term: 1 leader_uuid: "02d6b2ab7add458193cf60d80f814b44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "528e95197ba948cf9fbb1cdfae5f3def" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 46341 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa14e49f1dd243b6aa588d65fabe6c9a" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 33415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02d6b2ab7add458193cf60d80f814b44" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 36895 } health_report { overall_health: HEALTHY } } }
I20260925 07:32:44.992882 22746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:44.989872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42082 (local address 127.9.66.2:36895)
0925 07:32:44.989939 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 07:32:44.989940 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 07:32:44.989946 (+ 6us) server_negotiation.cc:407] Connection header received
0925 07:32:44.989975 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:32:44.989976 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:32:44.989995 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:32:44.990066 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 07:32:44.990228 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:32:44.990838 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:32:44.991970 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:32:44.992260 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:32:44.992329 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:32:44.992520 (+ 191us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 07:32:44.992634 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 07:32:44.992725 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 07:32:44.992793 (+ 68us) server_negotiation.cc:299] Negotiation successful
0925 07:32:44.992815 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 07:32:44.997301 22667 raft_consensus.cc:1261] T 027f81f3fa784fec965261678c71a01a P fa14e49f1dd243b6aa588d65fabe6c9a [term 1 FOLLOWER]: Refusing update from remote peer 02d6b2ab7add458193cf60d80f814b44: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:32:44.997648 22760 consensus_queue.cc:1035] T 027f81f3fa784fec965261678c71a01a P 02d6b2ab7add458193cf60d80f814b44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa14e49f1dd243b6aa588d65fabe6c9a" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 33415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:32:45.002296 22400 raft_consensus.cc:1261] T 027f81f3fa784fec965261678c71a01a P 528e95197ba948cf9fbb1cdfae5f3def [term 1 FOLLOWER]: Refusing update from remote peer 02d6b2ab7add458193cf60d80f814b44: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:32:45.002588 22755 consensus_queue.cc:1035] T 027f81f3fa784fec965261678c71a01a P 02d6b2ab7add458193cf60d80f814b44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "528e95197ba948cf9fbb1cdfae5f3def" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 46341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:32:45.003984 22763 mvcc.cc:205] Tried to move back new op lower bound from 7333157130221080576 to 7333157130161549312. Current Snapshot: MvccSnapshot[applied={T|T < 7333157130221080576}]
I20260925 07:32:45.006392 22248 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59900:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 07:32:45.006503 22248 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59900:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 07:32:45.007251 22248 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d7c638cf5b1545c787782729be7883b5: Sending DeleteTablet for 3 replicas of tablet 027f81f3fa784fec965261678c71a01a
I20260925 07:32:45.007424 22764 mvcc.cc:205] Tried to move back new op lower bound from 7333157130221080576 to 7333157130161549312. Current Snapshot: MvccSnapshot[applied={T|T < 7333157130221080576}]
I20260925 07:32:45.007776 22514 tablet_service.cc:1514] Processing DeleteTablet for tablet 027f81f3fa784fec965261678c71a01a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:32:45 UTC) from {username='kudu', principal='kudu/127.9.66.62@KRBTEST.COM'} at 127.0.0.1:42072
I20260925 07:32:45.007792 22380 tablet_service.cc:1514] Processing DeleteTablet for tablet 027f81f3fa784fec965261678c71a01a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:32:45 UTC) from {username='kudu', principal='kudu/127.9.66.62@KRBTEST.COM'} at 127.0.0.1:50450
I20260925 07:32:45.008024 22648 tablet_service.cc:1514] Processing DeleteTablet for tablet 027f81f3fa784fec965261678c71a01a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:32:45 UTC) from {username='kudu', principal='kudu/127.9.66.62@KRBTEST.COM'} at 127.0.0.1:36252
I20260925 07:32:45.008220 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 22286
I20260925 07:32:45.009550 22771 tablet_replica.cc:330] T 027f81f3fa784fec965261678c71a01a P 02d6b2ab7add458193cf60d80f814b44: stopping tablet replica
I20260925 07:32:45.009649 22771 raft_consensus.cc:2229] T 027f81f3fa784fec965261678c71a01a P 02d6b2ab7add458193cf60d80f814b44 [term 1 LEADER]: Raft consensus shutting down.
I20260925 07:32:45.009732 22771 raft_consensus.cc:2258] T 027f81f3fa784fec965261678c71a01a P 02d6b2ab7add458193cf60d80f814b44 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:32:45.011152 22771 ts_tablet_manager.cc:1905] T 027f81f3fa784fec965261678c71a01a P 02d6b2ab7add458193cf60d80f814b44: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 07:32:45.011798 22773 tablet_replica.cc:330] T 027f81f3fa784fec965261678c71a01a P fa14e49f1dd243b6aa588d65fabe6c9a: stopping tablet replica
I20260925 07:32:45.011915 22773 raft_consensus.cc:2229] T 027f81f3fa784fec965261678c71a01a P fa14e49f1dd243b6aa588d65fabe6c9a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 07:32:45.012003 22773 raft_consensus.cc:2258] T 027f81f3fa784fec965261678c71a01a P fa14e49f1dd243b6aa588d65fabe6c9a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:32:45.013059 22771 ts_tablet_manager.cc:1918] T 027f81f3fa784fec965261678c71a01a P 02d6b2ab7add458193cf60d80f814b44: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 07:32:45.013319 22771 log.cc:1196] T 027f81f3fa784fec965261678c71a01a P 02d6b2ab7add458193cf60d80f814b44: Deleting WAL directory at /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/ts-1/wal/wals/027f81f3fa784fec965261678c71a01a
W20260925 07:32:45.016347 22470 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.66.1:46341 (error 108)
W20260925 07:32:45.016776 22235 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.66.1:46341 (error 108)
I20260925 07:32:45.017725 22742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:45.017611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.1:46341 (local address 127.0.0.1:50466)
0925 07:32:45.017660 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0925 07:32:45.017693 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.1:46341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260925 07:32:45.017838 22235 catalog_manager.cc:4584] TS 528e95197ba948cf9fbb1cdfae5f3def (127.9.66.1:46341): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 027f81f3fa784fec965261678c71a01a: Network error: Client connection negotiation failed: client connection to 127.9.66.1:46341: connect: Connection refused (error 111)
I20260925 07:32:45.018013 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 22450
I20260925 07:32:45.021981 22773 ts_tablet_manager.cc:1905] T 027f81f3fa784fec965261678c71a01a P fa14e49f1dd243b6aa588d65fabe6c9a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 07:32:45.023687 22773 ts_tablet_manager.cc:1918] T 027f81f3fa784fec965261678c71a01a P fa14e49f1dd243b6aa588d65fabe6c9a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 07:32:45.023763 22773 log.cc:1196] T 027f81f3fa784fec965261678c71a01a P fa14e49f1dd243b6aa588d65fabe6c9a: Deleting WAL directory at /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790321392278483-9480-0/minicluster-data/ts-2/wal/wals/027f81f3fa784fec965261678c71a01a
I20260925 07:32:45.024065 22773 ts_tablet_manager.cc:1939] T 027f81f3fa784fec965261678c71a01a P fa14e49f1dd243b6aa588d65fabe6c9a: Deleting consensus metadata
I20260925 07:32:45.024483 22232 catalog_manager.cc:4841] TS fa14e49f1dd243b6aa588d65fabe6c9a (127.9.66.3:33415): tablet 027f81f3fa784fec965261678c71a01a (table test-table [id=ce9dca80d988435db8ad619817b027d2]) successfully deleted
W20260925 07:32:45.029158 22742 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:32:45.028816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.2:36895 (local address 127.0.0.1:42094)
0925 07:32:45.028862 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0925 07:32:45.028872 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 07:32:45.028908 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:32:45.029111 (+ 203us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.2:36895: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":29}
I20260925 07:32:45.029583 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 22583
I20260925 07:32:45.034624 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 22215
I20260925 07:32:45.040881 9480 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260925 07:32:45.385819 9480 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260925 07:32:45.385936 9480 mini_ranger.cc:66] Stopping Ranger...
I20260925 07:32:45.722110 9480 mini_ranger.cc:68] Stopped Ranger
2026-09-25 07:32:45.722 UTC [21282] LOG: received smart shutdown request
2026-09-25 07:32:45.726 UTC [21282] LOG: background worker "logical replication launcher" (PID 21291) exited with exit code 1
2026-09-25 07:32:45.726 UTC [21286] LOG: shutting down
2026-09-25 07:32:45.882 UTC [21282] LOG: database system is shut down
2026-09-25T07:32:45Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (46032 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790321392278483-9480-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 07:32:46 dist-test-slave-57fn krb5kdc[22784](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 07:32:46 dist-test-slave-57fn krb5kdc[22784](info): set up 2 sockets
Sep 25 07:32:46 dist-test-slave-57fn krb5kdc[22784](info): commencing operation
krb5kdc: starting...
W20260925 07:32:47.013382 9480 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790321392278483-9480-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790321392278483-9480-0/krb5kdc/test-user.keytab.
Sep 25 07:32:47 dist-test-slave-57fn krb5kdc[22784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321567, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T07:32:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T07:32:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.66.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.66.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.66.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab.
Entry for principal kudu/127.9.66.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab.
I20260925 07:32:47.130306 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790321392278483-9480-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790321392278483-9480-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790321392278483-9480-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.66.62:39073
--webserver_interface=127.9.66.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab
--principal=kudu/127.9.66.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:37239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.66.62:39073,127.9.66.61:37987,127.9.66.60:34971
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:32:47.206851 22797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:32:47.207062 22797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:32:47.207090 22797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:32:47.208264 22797 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:32:47.208303 22797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:32:47.208317 22797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:32:47.208330 22797 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:32:47.208343 22797 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:32:47.209894 22797 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:37239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790321392278483-9480-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790321392278483-9480-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.66.62:39073,127.9.66.61:37987,127.9.66.60:34971
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab
--principal=kudu/127.9.66.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.66.62:39073
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.66.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22797
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790321392278483-9480-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:32:47.210155 22797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:32:47.210387 22797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:32:47.215131 22803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:32:47.215149 22802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:32:47.215278 22797 server_base.cc:1034] running on GCE node
W20260925 07:32:47.215169 22805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:32:47.215606 22797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:32:47.215898 22797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:32:47.217067 22797 hybrid_clock.cc:648] HybridClock initialized: now 1790321567217054 us; error 28 us; skew 500 ppm
Sep 25 07:32:47 dist-test-slave-57fn krb5kdc[22784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321567, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:32:47.219575 22797 init.cc:377] Logged in from keytab as kudu/127.9.66.62@KRBTEST.COM (short username kudu)
I20260925 07:32:47.220001 22797 webserver.cc:460] Webserver started at http://127.9.66.62:35265/ using document root <none> and password file <none>
I20260925 07:32:47.220222 22797 fs_manager.cc:359] Metadata directory not provided
I20260925 07:32:47.220289 22797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:32:47.220444 22797 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:32:47.221468 22797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790321392278483-9480-0/minicluster-data/master-0/data/instance:
uuid: "0fc18dd26bc34a0b8eac8010e3a9f734"
format_stamp: "Formatted at 2026-09-25 07:32:47 on dist-test-slave-57fn"
server_key: "0f28d39af4996cb50366d566773e2fa4"
server_key_iv: "2b3902558e3e3479fb72285882902a78"
server_key_version: "encryptionkey@0"
I20260925 07:32:47.221920 22797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790321392278483-9480-0/minicluster-data/master-0/wal/instance:
uuid: "0fc18dd26bc34a0b8eac8010e3a9f734"
format_stamp: "Formatted at 2026-09-25 07:32:47 on dist-test-slave-57fn"
server_key: "0f28d39af4996cb50366d566773e2fa4"
server_key_iv: "2b3902558e3e3479fb72285882902a78"
server_key_version: "encryptionkey@0"
I20260925 07:32:47.223281 22797 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:32:47.224152 22812 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:32:47.224393 22797 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:32:47.224501 22797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790321392278483-9480-0/minicluster-data/master-0/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790321392278483-9480-0/minicluster-data/master-0/wal
uuid: "0fc18dd26bc34a0b8eac8010e3a9f734"
format_stamp: "Formatted at 2026-09-25 07:32:47 on dist-test-slave-57fn"
server_key: "0f28d39af4996cb50366d566773e2fa4"
server_key_iv: "2b3902558e3e3479fb72285882902a78"
server_key_version: "encryptionkey@0"
I20260925 07:32:47.224578 22797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790321392278483-9480-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790321392278483-9480-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790321392278483-9480-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:32:47.241833 22797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:32:47.243588 22797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:32:47.243762 22797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:32:47.247536 22797 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.62:39073
I20260925 07:32:47.247571 22864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.62:39073 every 8 connection(s)
I20260925 07:32:47.247933 22797 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
I20260925 07:32:47.248745 22865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:32:47.250267 22865 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.66.62" port: 39073 } has no permanent_uuid. Determining permanent_uuid...
I20260925 07:32:47.254472 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 22797
I20260925 07:32:47.254611 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790321392278483-9480-0/minicluster-data/master-0/wal/instance
I20260925 07:32:47.254817 9480 external_mini_cluster.cc:1428] Setting key 2502f9b0deb3469f294cff4c5d14058e
Sep 25 07:32:47 dist-test-slave-57fn krb5kdc[22784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321567, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.62@KRBTEST.COM for kudu/127.9.66.62@KRBTEST.COM
I20260925 07:32:47.259852 22867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:47.250868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:39073 (local address 127.0.0.1:39018)
0925 07:32:47.251000 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0925 07:32:47.251031 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0925 07:32:47.251509 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:32:47.251779 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:32:47.251781 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:32:47.252051 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:32:47.252189 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:32:47.252193 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:32:47.253145 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:32:47.253146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:32:47.253919 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:32:47.253921 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:32:47.254415 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:32:47.254416 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:32:47.254476 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:32:47.255034 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:32:47.255041 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:32:47.256446 (+ 1405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:32:47.258435 (+ 1989us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:32:47.258441 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:32:47.258443 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:32:47.258629 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:32:47.258884 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:32:47.258884 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:32:47.258885 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:32:47.258970 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:32:47.259301 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:32:47.259305 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:32:47.259364 (+ 59us) client_negotiation.cc:769] Sending connection context
0925 07:32:47.259469 (+ 105us) client_negotiation.cc:240] Negotiation successful
0925 07:32:47.259694 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"mutex_wait_us":97,"thread_start_us":61,"threads_started":1}
I20260925 07:32:47.259856 22868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:47.251001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39018 (local address 127.9.66.62:39073)
0925 07:32:47.251106 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0925 07:32:47.251110 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 07:32:47.251140 (+ 30us) server_negotiation.cc:407] Connection header received
0925 07:32:47.251660 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:32:47.251665 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:32:47.251685 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:32:47.251775 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:32:47.252291 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:32:47.253040 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:32:47.254000 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:32:47.254328 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:32:47.254442 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:32:47.256600 (+ 2158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:32:47.256608 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:32:47.256608 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:32:47.256616 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:32:47.258351 (+ 1735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:32:47.258725 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:32:47.258728 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:32:47.258728 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:32:47.258805 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:32:47.259034 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:32:47.259035 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:32:47.259036 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:32:47.259228 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:32:47.259321 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 07:32:47.259494 (+ 173us) server_negotiation.cc:299] Negotiation successful
0925 07:32:47.259633 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79,"thread_start_us":37,"threads_started":1}
I20260925 07:32:47.260561 22865 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.66.61" port: 37987 } has no permanent_uuid. Determining permanent_uuid...
I20260925 07:32:47.260939 22867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:47.260841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.61:37987 (local address 127.0.0.1:41966)
0925 07:32:47.260885 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0925 07:32:47.260922 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.61:37987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260925 07:32:47.261106 22813 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.66.61:37987: connect: Connection refused (error 111)
I20260925 07:32:47.261605 22867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:47.261451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.61:37987 (local address 127.0.0.1:41974)
0925 07:32:47.261552 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0925 07:32:47.261579 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.61:37987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
W20260925 07:32:47.261723 22865 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.66.61:37987: Network error: Client connection negotiation failed: client connection to 127.9.66.61:37987: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.9.66.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.66.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.66.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.61.keytab.
Entry for principal kudu/127.9.66.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.61.keytab.
I20260925 07:32:47.280311 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790321392278483-9480-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790321392278483-9480-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790321392278483-9480-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790321392278483-9480-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.66.61:37987
--webserver_interface=127.9.66.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.61.keytab
--principal=kudu/127.9.66.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:37239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.66.62:39073,127.9.66.61:37987,127.9.66.60:34971
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 07:32:47.310936 22865 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.66.61" port: 37987 } attempt: 1
I20260925 07:32:47.311465 22867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:47.311294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.61:37987 (local address 127.0.0.1:41980)
0925 07:32:47.311371 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0925 07:32:47.311426 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.61:37987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260925 07:32:47.311851 22867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:47.311752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.61:37987 (local address 127.0.0.1:41984)
0925 07:32:47.311798 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0925 07:32:47.311828 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.61:37987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260925 07:32:47.311959 22865 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.66.61:37987: Network error: Client connection negotiation failed: client connection to 127.9.66.61:37987: connect: Connection refused (error 111)
W20260925 07:32:47.362644 22872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:32:47.362855 22872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:32:47.362879 22872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:32:47.364020 22872 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:32:47.364069 22872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:32:47.364084 22872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:32:47.364096 22872 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:32:47.364108 22872 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:32:47.365902 22872 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:37239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790321392278483-9480-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790321392278483-9480-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.9.66.62:39073,127.9.66.61:37987,127.9.66.60:34971
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.61.keytab
--principal=kudu/127.9.66.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.66.61:37987
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790321392278483-9480-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.9.66.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22872
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790321392278483-9480-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:32:47.366124 22872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:32:47.366298 22872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:32:47.371089 22877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:32:47.371096 22878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:32:47.371132 22880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:32:47.371264 22872 server_base.cc:1034] running on GCE node
I20260925 07:32:47.371570 22872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:32:47.371816 22872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:32:47.372959 22872 hybrid_clock.cc:648] HybridClock initialized: now 1790321567372942 us; error 40 us; skew 500 ppm
Sep 25 07:32:47 dist-test-slave-57fn krb5kdc[22784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321567, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:32:47.375475 22872 init.cc:377] Logged in from keytab as kudu/127.9.66.61@KRBTEST.COM (short username kudu)
I20260925 07:32:47.375950 22872 webserver.cc:460] Webserver started at http://127.9.66.61:36365/ using document root <none> and password file <none>
I20260925 07:32:47.376186 22872 fs_manager.cc:359] Metadata directory not provided
I20260925 07:32:47.376255 22872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:32:47.376405 22872 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:32:47.377452 22872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790321392278483-9480-0/minicluster-data/master-1/data/instance:
uuid: "61cc89ef8d0f4893a84cd180cd92c209"
format_stamp: "Formatted at 2026-09-25 07:32:47 on dist-test-slave-57fn"
server_key: "7042e315fdb94c005e126e26f23f4c72"
server_key_iv: "cf2e1048f383bc760bd7119dc8f7882a"
server_key_version: "encryptionkey@0"
I20260925 07:32:47.377925 22872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790321392278483-9480-0/minicluster-data/master-1/wal/instance:
uuid: "61cc89ef8d0f4893a84cd180cd92c209"
format_stamp: "Formatted at 2026-09-25 07:32:47 on dist-test-slave-57fn"
server_key: "7042e315fdb94c005e126e26f23f4c72"
server_key_iv: "cf2e1048f383bc760bd7119dc8f7882a"
server_key_version: "encryptionkey@0"
I20260925 07:32:47.379240 22872 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:32:47.379997 22887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:32:47.380203 22872 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:32:47.380165 22865 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.66.61" port: 37987 } attempt: 2
I20260925 07:32:47.380306 22872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790321392278483-9480-0/minicluster-data/master-1/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790321392278483-9480-0/minicluster-data/master-1/wal
uuid: "61cc89ef8d0f4893a84cd180cd92c209"
format_stamp: "Formatted at 2026-09-25 07:32:47 on dist-test-slave-57fn"
server_key: "7042e315fdb94c005e126e26f23f4c72"
server_key_iv: "cf2e1048f383bc760bd7119dc8f7882a"
server_key_version: "encryptionkey@0"
I20260925 07:32:47.380403 22872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790321392278483-9480-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790321392278483-9480-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790321392278483-9480-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:32:47.380651 22867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:47.380508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.61:37987 (local address 127.0.0.1:42000)
0925 07:32:47.380572 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0925 07:32:47.380614 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.61:37987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260925 07:32:47.381038 22867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:47.380941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.61:37987 (local address 127.0.0.1:42008)
0925 07:32:47.380990 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0925 07:32:47.381016 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.61:37987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260925 07:32:47.381165 22865 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.66.61:37987: Network error: Client connection negotiation failed: client connection to 127.9.66.61:37987: connect: Connection refused (error 111)
I20260925 07:32:47.394270 22872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:32:47.396502 22872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:32:47.396623 22872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:32:47.400601 22872 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.61:37987
I20260925 07:32:47.400660 22939 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.61:37987 every 8 connection(s)
I20260925 07:32:47.400983 22872 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790321392278483-9480-0/minicluster-data/master-1/data/info.pb
I20260925 07:32:47.401791 22940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:32:47.403378 22940 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.66.62" port: 39073 } has no permanent_uuid. Determining permanent_uuid...
I20260925 07:32:47.405177 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 22872
I20260925 07:32:47.405277 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790321392278483-9480-0/minicluster-data/master-1/wal/instance
I20260925 07:32:47.405452 9480 external_mini_cluster.cc:1428] Setting key 5a68c93fd793662a7438440cd8156658
Sep 25 07:32:47 dist-test-slave-57fn krb5kdc[22784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321567, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.61@KRBTEST.COM for kudu/127.9.66.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.66.60@KRBTEST.COM; defaulting to no policy
I20260925 07:32:47.414530 22868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:47.404056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39020 (local address 127.9.66.62:39073)
0925 07:32:47.404122 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 07:32:47.404125 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:32:47.404339 (+ 214us) server_negotiation.cc:407] Connection header received
0925 07:32:47.404770 (+ 431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:32:47.404771 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:32:47.404784 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:32:47.404837 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:32:47.405681 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:32:47.406334 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:32:47.407492 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:32:47.407705 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:32:47.407797 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:32:47.410121 (+ 2324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:32:47.410122 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:32:47.410123 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:32:47.410129 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:32:47.412104 (+ 1975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:32:47.412694 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:32:47.412696 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:32:47.412696 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:32:47.412766 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:32:47.413093 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:32:47.413094 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:32:47.413095 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:32:47.413259 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:32:47.413308 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 07:32:47.414355 (+ 1047us) server_negotiation.cc:299] Negotiation successful
0925 07:32:47.414440 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 07:32:47.414865 22942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:47.404038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:39073 (local address 127.0.0.1:39020)
0925 07:32:47.404228 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0925 07:32:47.404261 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 07:32:47.404681 (+ 420us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:32:47.404909 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:32:47.404912 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:32:47.405268 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:32:47.405591 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:32:47.405598 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:32:47.406426 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:32:47.406427 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:32:47.407398 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:32:47.407400 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:32:47.407819 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:32:47.407821 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:32:47.407942 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:32:47.408442 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:32:47.408450 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:32:47.410008 (+ 1558us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:32:47.412211 (+ 2203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:32:47.412217 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:32:47.412219 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:32:47.412590 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:32:47.412849 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:32:47.412850 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:32:47.412851 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:32:47.412997 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:32:47.413978 (+ 981us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:32:47.413984 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:32:47.414180 (+ 196us) client_negotiation.cc:769] Sending connection context
0925 07:32:47.414331 (+ 151us) client_negotiation.cc:240] Negotiation successful
0925 07:32:47.414485 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":78,"threads_started":1}
I20260925 07:32:47.415544 22940 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.66.61" port: 37987 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.66.60@KRBTEST.COM" created.
Sep 25 07:32:47 dist-test-slave-57fn krb5kdc[22784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321567, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.61@KRBTEST.COM for kudu/127.9.66.61@KRBTEST.COM
I20260925 07:32:47.424623 22942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:47.415865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.61:37987 (local address 127.0.0.1:42018)
0925 07:32:47.416133 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0925 07:32:47.416145 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 07:32:47.416203 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:32:47.416362 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:32:47.416363 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:32:47.416549 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:32:47.416631 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:32:47.416633 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:32:47.418216 (+ 1583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:32:47.418217 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:32:47.419131 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:32:47.419133 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:32:47.419808 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:32:47.419809 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:32:47.419881 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:32:47.420344 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:32:47.420349 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:32:47.421495 (+ 1146us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:32:47.423656 (+ 2161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:32:47.423657 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:32:47.423658 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:32:47.423813 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:32:47.424026 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:32:47.424026 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:32:47.424027 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:32:47.424075 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:32:47.424403 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:32:47.424403 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:32:47.424436 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 07:32:47.424485 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 07:32:47.424554 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257}
I20260925 07:32:47.424782 22944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:47.415971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42018 (local address 127.9.66.61:37987)
0925 07:32:47.416150 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 07:32:47.416155 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 07:32:47.416201 (+ 46us) server_negotiation.cc:407] Connection header received
0925 07:32:47.416262 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:32:47.416266 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:32:47.416279 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:32:47.416360 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:32:47.416731 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:32:47.418118 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:32:47.419224 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:32:47.419729 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:32:47.419801 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:32:47.421623 (+ 1822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:32:47.421630 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:32:47.421631 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:32:47.421638 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:32:47.423564 (+ 1926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:32:47.423888 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:32:47.423891 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:32:47.423892 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:32:47.423960 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:32:47.424136 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:32:47.424137 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:32:47.424137 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:32:47.424335 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:32:47.424425 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 07:32:47.424521 (+ 96us) server_negotiation.cc:299] Negotiation successful
0925 07:32:47.424636 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":8,"server-negotiator.queue_time_us":140,"thread_start_us":109,"threads_started":1}
I20260925 07:32:47.425379 22940 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.66.60" port: 34971 } has no permanent_uuid. Determining permanent_uuid...
I20260925 07:32:47.425786 22942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:47.425640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.60:34971 (local address 127.0.0.1:33048)
0925 07:32:47.425696 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0925 07:32:47.425752 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.60:34971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260925 07:32:47.425963 22889 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.66.60:34971: connect: Connection refused (error 111)
I20260925 07:32:47.426393 22942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:47.426287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.60:34971 (local address 127.0.0.1:33050)
0925 07:32:47.426335 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0925 07:32:47.426363 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.60:34971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260925 07:32:47.426544 22940 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.66.60:34971: Network error: Client connection negotiation failed: client connection to 127.9.66.60:34971: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.66.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.60.keytab.
Entry for principal kudu/127.9.66.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.60.keytab.
I20260925 07:32:47.441778 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790321392278483-9480-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790321392278483-9480-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790321392278483-9480-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790321392278483-9480-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.66.60:34971
--webserver_interface=127.9.66.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.60.keytab
--principal=kudu/127.9.66.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:37239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.66.62:39073,127.9.66.61:37987,127.9.66.60:34971
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 07:32:47.472339 22865 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.66.61" port: 37987 } attempt: 3
I20260925 07:32:47.475781 22940 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.66.60" port: 34971 } attempt: 1
I20260925 07:32:47.476193 22942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:47.476088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.60:34971 (local address 127.0.0.1:33056)
0925 07:32:47.476132 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0925 07:32:47.476161 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.60:34971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260925 07:32:47.476917 22942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:47.476842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.60:34971 (local address 127.0.0.1:33058)
0925 07:32:47.476874 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0925 07:32:47.476894 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.60:34971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260925 07:32:47.477065 22940 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.66.60:34971: Network error: Client connection negotiation failed: client connection to 127.9.66.60:34971: connect: Connection refused (error 111)
Sep 25 07:32:47 dist-test-slave-57fn krb5kdc[22784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321567, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.62@KRBTEST.COM for kudu/127.9.66.61@KRBTEST.COM
I20260925 07:32:47.481870 22867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:47.472722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.61:37987 (local address 127.0.0.1:42034)
0925 07:32:47.472783 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0925 07:32:47.472796 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 07:32:47.472841 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:32:47.473058 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:32:47.473059 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:32:47.473246 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:32:47.473323 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:32:47.473325 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:32:47.474463 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:32:47.474465 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:32:47.475291 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:32:47.475294 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:32:47.475939 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:32:47.475941 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:32:47.476055 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:32:47.476545 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:32:47.476551 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:32:47.478208 (+ 1657us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:32:47.480555 (+ 2347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:32:47.480556 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:32:47.480556 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:32:47.480806 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:32:47.481108 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:32:47.481109 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:32:47.481109 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:32:47.481183 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:32:47.481487 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:32:47.481488 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:32:47.481566 (+ 78us) client_negotiation.cc:769] Sending connection context
0925 07:32:47.481650 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 07:32:47.481758 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260925 07:32:47.481921 22944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:47.472806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42034 (local address 127.9.66.61:37987)
0925 07:32:47.472862 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 07:32:47.472864 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:32:47.472877 (+ 13us) server_negotiation.cc:407] Connection header received
0925 07:32:47.472923 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:32:47.472925 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:32:47.472941 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:32:47.473042 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:32:47.473420 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:32:47.474341 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:32:47.475380 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:32:47.475765 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:32:47.475851 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:32:47.478441 (+ 2590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:32:47.478443 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:32:47.478443 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:32:47.478452 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:32:47.480450 (+ 1998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:32:47.480903 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:32:47.480904 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:32:47.480905 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:32:47.481022 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:32:47.481257 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:32:47.481258 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:32:47.481259 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:32:47.481411 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:32:47.481483 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 07:32:47.481689 (+ 206us) server_negotiation.cc:299] Negotiation successful
0925 07:32:47.481784 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260925 07:32:47.482580 22865 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.66.60" port: 34971 } has no permanent_uuid. Determining permanent_uuid...
I20260925 07:32:47.482957 22867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:47.482824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.60:34971 (local address 127.0.0.1:33062)
0925 07:32:47.482885 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0925 07:32:47.482918 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.60:34971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260925 07:32:47.483354 22867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:47.483233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.60:34971 (local address 127.0.0.1:33064)
0925 07:32:47.483301 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0925 07:32:47.483329 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.60:34971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260925 07:32:47.483489 22865 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.66.60:34971: Network error: Client connection negotiation failed: client connection to 127.9.66.60:34971: connect: Connection refused (error 111)
I20260925 07:32:47.514631 22865 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.66.60" port: 34971 } attempt: 1
I20260925 07:32:47.515187 22867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:47.515000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.60:34971 (local address 127.0.0.1:33080)
0925 07:32:47.515079 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0925 07:32:47.515129 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.60:34971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260925 07:32:47.515693 22867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:47.515475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.60:34971 (local address 127.0.0.1:33086)
0925 07:32:47.515629 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 07:32:47.515656 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.60:34971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":107,"threads_started":1}
W20260925 07:32:47.515832 22865 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.66.60:34971: Network error: Client connection negotiation failed: client connection to 127.9.66.60:34971: connect: Connection refused (error 111)
W20260925 07:32:47.527570 22947 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:32:47.527765 22947 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:32:47.527788 22947 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:32:47.529161 22947 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:32:47.529206 22947 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:32:47.529219 22947 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:32:47.529232 22947 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:32:47.529243 22947 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:32:47.530763 22947 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:37239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790321392278483-9480-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790321392278483-9480-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.9.66.62:39073,127.9.66.61:37987,127.9.66.60:34971
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.60.keytab
--principal=kudu/127.9.66.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.66.60:34971
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790321392278483-9480-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.9.66.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22947
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790321392278483-9480-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:32:47.530956 22947 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:32:47.531143 22947 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:32:47.535985 22953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:32:47.535984 22954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:32:47.536049 22947 server_base.cc:1034] running on GCE node
W20260925 07:32:47.535985 22956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:32:47.536485 22947 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:32:47.536760 22947 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:32:47.537894 22947 hybrid_clock.cc:648] HybridClock initialized: now 1790321567537884 us; error 25 us; skew 500 ppm
Sep 25 07:32:47 dist-test-slave-57fn krb5kdc[22784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321567, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:32:47.540318 22947 init.cc:377] Logged in from keytab as kudu/127.9.66.60@KRBTEST.COM (short username kudu)
I20260925 07:32:47.540742 22947 webserver.cc:460] Webserver started at http://127.9.66.60:36975/ using document root <none> and password file <none>
I20260925 07:32:47.540951 22947 fs_manager.cc:359] Metadata directory not provided
I20260925 07:32:47.541014 22947 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:32:47.541240 22947 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:32:47.542376 22947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790321392278483-9480-0/minicluster-data/master-2/data/instance:
uuid: "024b1bbe80a4470b8090af8bc0ec3b59"
format_stamp: "Formatted at 2026-09-25 07:32:47 on dist-test-slave-57fn"
server_key: "85da29d7e8f98ebe97cde902ef6a23b0"
server_key_iv: "0347cdb6b81d5fe186152e6968126968"
server_key_version: "encryptionkey@0"
I20260925 07:32:47.542739 22947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790321392278483-9480-0/minicluster-data/master-2/wal/instance:
uuid: "024b1bbe80a4470b8090af8bc0ec3b59"
format_stamp: "Formatted at 2026-09-25 07:32:47 on dist-test-slave-57fn"
server_key: "85da29d7e8f98ebe97cde902ef6a23b0"
server_key_iv: "0347cdb6b81d5fe186152e6968126968"
server_key_version: "encryptionkey@0"
I20260925 07:32:47.544055 22947 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 07:32:47.544796 22963 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:32:47.544996 22947 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:32:47.545104 22947 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790321392278483-9480-0/minicluster-data/master-2/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790321392278483-9480-0/minicluster-data/master-2/wal
uuid: "024b1bbe80a4470b8090af8bc0ec3b59"
format_stamp: "Formatted at 2026-09-25 07:32:47 on dist-test-slave-57fn"
server_key: "85da29d7e8f98ebe97cde902ef6a23b0"
server_key_iv: "0347cdb6b81d5fe186152e6968126968"
server_key_version: "encryptionkey@0"
I20260925 07:32:47.545190 22947 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790321392278483-9480-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790321392278483-9480-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790321392278483-9480-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:32:47.545200 22940 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.66.60" port: 34971 } attempt: 2
I20260925 07:32:47.545755 22942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:47.545609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.60:34971 (local address 127.0.0.1:33088)
0925 07:32:47.545671 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0925 07:32:47.545712 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.60:34971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260925 07:32:47.546132 22942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:47.546053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.60:34971 (local address 127.0.0.1:33094)
0925 07:32:47.546078 (+ 25us) negotiation.cc:107] Waiting for socket to connect
0925 07:32:47.546108 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.60:34971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13}
W20260925 07:32:47.546252 22940 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.66.60:34971: Network error: Client connection negotiation failed: client connection to 127.9.66.60:34971: connect: Connection refused (error 111)
I20260925 07:32:47.555920 22947 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:32:47.558035 22947 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:32:47.558197 22947 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:32:47.562239 22947 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.60:34971
I20260925 07:32:47.562287 23015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.60:34971 every 8 connection(s)
I20260925 07:32:47.562624 22947 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790321392278483-9480-0/minicluster-data/master-2/data/info.pb
I20260925 07:32:47.563434 23016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:32:47.564842 23016 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.66.62" port: 39073 } has no permanent_uuid. Determining permanent_uuid...
I20260925 07:32:47.566407 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 22947
I20260925 07:32:47.566511 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790321392278483-9480-0/minicluster-data/master-2/wal/instance
I20260925 07:32:47.566707 9480 external_mini_cluster.cc:1428] Setting key aff003fdc2d3a494bde7c328c540099a
Sep 25 07:32:47 dist-test-slave-57fn krb5kdc[22784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321567, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.60@KRBTEST.COM for kudu/127.9.66.62@KRBTEST.COM
Sep 25 07:32:47 dist-test-slave-57fn krb5kdc[22784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321567, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.66.60@KRBTEST.COM
I20260925 07:32:47.574155 22868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:47.565452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39032 (local address 127.9.66.62:39073)
0925 07:32:47.565558 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0925 07:32:47.565561 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:32:47.565707 (+ 146us) server_negotiation.cc:407] Connection header received
0925 07:32:47.566146 (+ 439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:32:47.566147 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:32:47.566156 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:32:47.566207 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:32:47.567059 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:32:47.567723 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:32:47.568688 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:32:47.568885 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:32:47.568960 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:32:47.571018 (+ 2058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:32:47.571020 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:32:47.571020 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:32:47.571028 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:32:47.572653 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:32:47.573148 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:32:47.573149 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:32:47.573150 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:32:47.573207 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:32:47.573452 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:32:47.573453 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:32:47.573454 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:32:47.573622 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:32:47.573699 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 07:32:47.573968 (+ 269us) server_negotiation.cc:299] Negotiation successful
0925 07:32:47.574046 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 07:32:47.574308 23018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:47.565395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:39073 (local address 127.0.0.1:39032)
0925 07:32:47.565591 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0925 07:32:47.565630 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 07:32:47.566064 (+ 434us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:32:47.566288 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:32:47.566292 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:32:47.566663 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:32:47.566939 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:32:47.566944 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:32:47.567818 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:32:47.567819 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:32:47.568588 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:32:47.568590 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:32:47.568967 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:32:47.568968 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:32:47.569098 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:32:47.569625 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:32:47.569633 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:32:47.570928 (+ 1295us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:32:47.572736 (+ 1808us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:32:47.572742 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:32:47.572744 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:32:47.573049 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:32:47.573274 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:32:47.573275 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:32:47.573275 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:32:47.573383 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:32:47.573690 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:32:47.573697 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:32:47.573831 (+ 134us) client_negotiation.cc:769] Sending connection context
0925 07:32:47.573933 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 07:32:47.574078 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":82,"threads_started":1}
I20260925 07:32:47.574922 23016 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.66.61" port: 37987 } has no permanent_uuid. Determining permanent_uuid...
I20260925 07:32:47.575747 23020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:47.567070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33100 (local address 127.9.66.60:34971)
0925 07:32:47.567270 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0925 07:32:47.567274 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 07:32:47.567297 (+ 23us) server_negotiation.cc:407] Connection header received
0925 07:32:47.567322 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:32:47.567324 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:32:47.567342 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:32:47.567429 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:32:47.568078 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:32:47.568794 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:32:47.569717 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:32:47.570041 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:32:47.570145 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:32:47.572808 (+ 2663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:32:47.572812 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:32:47.572813 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:32:47.572819 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:32:47.574498 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:32:47.574871 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:32:47.574874 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:32:47.574875 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:32:47.574950 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:32:47.575174 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:32:47.575175 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:32:47.575175 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:32:47.575342 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:32:47.575460 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0925 07:32:47.575604 (+ 144us) server_negotiation.cc:299] Negotiation successful
0925 07:32:47.575683 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":140,"threads_started":1}
Sep 25 07:32:47 dist-test-slave-57fn krb5kdc[22784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321567, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.60@KRBTEST.COM for kudu/127.9.66.61@KRBTEST.COM
Sep 25 07:32:47 dist-test-slave-57fn krb5kdc[22784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321567, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.66.62@KRBTEST.COM
I20260925 07:32:47.581771 23018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:47.575171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.61:37987 (local address 127.0.0.1:42036)
0925 07:32:47.575202 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0925 07:32:47.575215 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 07:32:47.575250 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:32:47.575511 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:32:47.575511 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:32:47.575691 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:32:47.575758 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:32:47.575760 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:32:47.576365 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:32:47.576366 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:32:47.577017 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:32:47.577019 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:32:47.577491 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:32:47.577492 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:32:47.577627 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:32:47.578007 (+ 380us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:32:47.578011 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:32:47.579288 (+ 1277us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:32:47.580774 (+ 1486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:32:47.580775 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:32:47.580775 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:32:47.580935 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:32:47.581155 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:32:47.581156 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:32:47.581156 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:32:47.581218 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:32:47.581461 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:32:47.581462 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:32:47.581503 (+ 41us) client_negotiation.cc:769] Sending connection context
0925 07:32:47.581598 (+ 95us) client_negotiation.cc:240] Negotiation successful
0925 07:32:47.581686 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16}
I20260925 07:32:47.581802 22944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:47.575299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42036 (local address 127.9.66.61:37987)
0925 07:32:47.575346 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 07:32:47.575366 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 07:32:47.575372 (+ 6us) server_negotiation.cc:407] Connection header received
0925 07:32:47.575407 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:32:47.575409 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:32:47.575419 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:32:47.575466 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:32:47.575824 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:32:47.576283 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:32:47.577114 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:32:47.577394 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:32:47.577463 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:32:47.579369 (+ 1906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:32:47.579371 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:32:47.579371 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:32:47.579379 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:32:47.580665 (+ 1286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:32:47.581019 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:32:47.581020 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:32:47.581020 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:32:47.581072 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:32:47.581279 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:32:47.581280 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:32:47.581281 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:32:47.581394 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:32:47.581469 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 07:32:47.581639 (+ 170us) server_negotiation.cc:299] Negotiation successful
0925 07:32:47.581714 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 07:32:47.582713 23016 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.66.60" port: 34971 } has no permanent_uuid. Determining permanent_uuid...
I20260925 07:32:47.583845 22868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:47.576683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39040 (local address 127.9.66.62:39073)
0925 07:32:47.576721 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 07:32:47.576722 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 07:32:47.576731 (+ 9us) server_negotiation.cc:407] Connection header received
0925 07:32:47.576751 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:32:47.576751 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:32:47.576759 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:32:47.576820 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:32:47.577192 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:32:47.577772 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:32:47.578451 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:32:47.578753 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:32:47.578822 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:32:47.581084 (+ 2262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:32:47.581086 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:32:47.581086 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:32:47.581093 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:32:47.582673 (+ 1580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:32:47.583088 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:32:47.583089 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:32:47.583089 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:32:47.583148 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:32:47.583350 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:32:47.583351 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:32:47.583351 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:32:47.583471 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:32:47.583644 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0925 07:32:47.583681 (+ 37us) server_negotiation.cc:299] Negotiation successful
0925 07:32:47.583754 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
Sep 25 07:32:47 dist-test-slave-57fn krb5kdc[22784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321567, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.60@KRBTEST.COM for kudu/127.9.66.60@KRBTEST.COM
Sep 25 07:32:47 dist-test-slave-57fn krb5kdc[22784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321567, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.66.61@KRBTEST.COM
I20260925 07:32:47.589908 23018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:47.582994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.60:34971 (local address 127.0.0.1:33104)
0925 07:32:47.583039 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0925 07:32:47.583050 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 07:32:47.583086 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:32:47.583203 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:32:47.583204 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:32:47.583306 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:32:47.583374 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:32:47.583375 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:32:47.584258 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:32:47.584259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:32:47.585017 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:32:47.585020 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:32:47.585458 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:32:47.585459 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:32:47.585501 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:32:47.585969 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:32:47.585973 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:32:47.587258 (+ 1285us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:32:47.588808 (+ 1550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:32:47.588809 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:32:47.588809 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:32:47.589015 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:32:47.589239 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:32:47.589239 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:32:47.589240 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:32:47.589305 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:32:47.589622 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:32:47.589623 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:32:47.589665 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 07:32:47.589737 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 07:32:47.589825 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260925 07:32:47.590044 23020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:47.583055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33104 (local address 127.9.66.60:34971)
0925 07:32:47.583099 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 07:32:47.583101 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:32:47.583107 (+ 6us) server_negotiation.cc:407] Connection header received
0925 07:32:47.583129 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:32:47.583130 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:32:47.583139 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:32:47.583202 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:32:47.583436 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:32:47.584180 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:32:47.585100 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:32:47.585365 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:32:47.585433 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:32:47.587351 (+ 1918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:32:47.587352 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:32:47.587353 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:32:47.587359 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:32:47.588726 (+ 1367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:32:47.589112 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:32:47.589113 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:32:47.589113 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:32:47.589164 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:32:47.589393 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:32:47.589394 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:32:47.589395 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:32:47.589548 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:32:47.589615 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 07:32:47.589811 (+ 196us) server_negotiation.cc:299] Negotiation successful
0925 07:32:47.589909 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 07:32:47.590996 22865 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.66.60" port: 34971 } attempt: 2
I20260925 07:32:47.591673 23016 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 024b1bbe80a4470b8090af8bc0ec3b59: Bootstrap starting.
I20260925 07:32:47.592442 22944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:47.584474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42040 (local address 127.9.66.61:37987)
0925 07:32:47.584509 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0925 07:32:47.584511 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:32:47.584570 (+ 59us) server_negotiation.cc:407] Connection header received
0925 07:32:47.584637 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:32:47.584637 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:32:47.584644 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:32:47.584685 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:32:47.585203 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:32:47.585779 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:32:47.586500 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:32:47.586674 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:32:47.586721 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:32:47.589285 (+ 2564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:32:47.589286 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:32:47.589286 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:32:47.589291 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:32:47.591279 (+ 1988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:32:47.591682 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:32:47.591683 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:32:47.591683 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:32:47.591738 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:32:47.591935 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:32:47.591935 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:32:47.591936 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:32:47.592044 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:32:47.592214 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0925 07:32:47.592262 (+ 48us) server_negotiation.cc:299] Negotiation successful
0925 07:32:47.592331 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 07:32:47.592903 23016 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 024b1bbe80a4470b8090af8bc0ec3b59: Neither blocks nor log segments found. Creating new log.
I20260925 07:32:47.593358 23016 log.cc:824] T 00000000000000000000000000000000 P 024b1bbe80a4470b8090af8bc0ec3b59: Log is configured to *not* fsync() on all Append() calls
I20260925 07:32:47.593822 9480 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 07:32:47.594440 23016 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 024b1bbe80a4470b8090af8bc0ec3b59: No bootstrap required, opened a new log
Sep 25 07:32:47 dist-test-slave-57fn krb5kdc[22784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321567, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.62@KRBTEST.COM for kudu/127.9.66.60@KRBTEST.COM
I20260925 07:32:47.596954 23016 raft_consensus.cc:348] T 00000000000000000000000000000000 P 024b1bbe80a4470b8090af8bc0ec3b59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fc18dd26bc34a0b8eac8010e3a9f734" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 39073 } } peers { permanent_uuid: "61cc89ef8d0f4893a84cd180cd92c209" member_type: VOTER last_known_addr { host: "127.9.66.61" port: 37987 } } peers { permanent_uuid: "024b1bbe80a4470b8090af8bc0ec3b59" member_type: VOTER last_known_addr { host: "127.9.66.60" port: 34971 } }
I20260925 07:32:47.597195 23016 raft_consensus.cc:374] T 00000000000000000000000000000000 P 024b1bbe80a4470b8090af8bc0ec3b59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:32:47.597288 23016 raft_consensus.cc:729] T 00000000000000000000000000000000 P 024b1bbe80a4470b8090af8bc0ec3b59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 024b1bbe80a4470b8090af8bc0ec3b59, State: Initialized, Role: FOLLOWER
I20260925 07:32:47.597512 23016 consensus_queue.cc:260] T 00000000000000000000000000000000 P 024b1bbe80a4470b8090af8bc0ec3b59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fc18dd26bc34a0b8eac8010e3a9f734" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 39073 } } peers { permanent_uuid: "61cc89ef8d0f4893a84cd180cd92c209" member_type: VOTER last_known_addr { host: "127.9.66.61" port: 37987 } } peers { permanent_uuid: "024b1bbe80a4470b8090af8bc0ec3b59" member_type: VOTER last_known_addr { host: "127.9.66.60" port: 34971 } }
I20260925 07:32:47.597975 23016 sys_catalog.cc:564] T 00000000000000000000000000000000 P 024b1bbe80a4470b8090af8bc0ec3b59 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:32:47.598855 22867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:47.591253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.60:34971 (local address 127.0.0.1:33112)
0925 07:32:47.591303 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0925 07:32:47.591313 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 07:32:47.591391 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:32:47.591540 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:32:47.591541 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:32:47.591705 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:32:47.591749 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:32:47.591751 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:32:47.592661 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:32:47.592662 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:32:47.593438 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:32:47.593441 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:32:47.593929 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:32:47.593930 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:32:47.593988 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:32:47.594387 (+ 399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:32:47.594392 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:32:47.595707 (+ 1315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:32:47.597598 (+ 1891us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:32:47.597599 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:32:47.597599 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:32:47.597842 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:32:47.598075 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:32:47.598076 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:32:47.598076 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:32:47.598140 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:32:47.598502 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:32:47.598503 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:32:47.598555 (+ 52us) client_negotiation.cc:769] Sending connection context
0925 07:32:47.598634 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 07:32:47.598742 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260925 07:32:47.599476 23020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:47.591292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33112 (local address 127.9.66.60:34971)
0925 07:32:47.591327 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0925 07:32:47.591328 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 07:32:47.591339 (+ 11us) server_negotiation.cc:407] Connection header received
0925 07:32:47.591464 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:32:47.591465 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:32:47.591474 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:32:47.591523 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:32:47.591843 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:32:47.592537 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:32:47.593567 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:32:47.593837 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:32:47.593929 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:32:47.595806 (+ 1877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:32:47.595808 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:32:47.595808 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:32:47.595815 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:32:47.597474 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:32:47.597936 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:32:47.597937 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:32:47.597937 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:32:47.597996 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:32:47.598227 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:32:47.598227 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:32:47.598228 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:32:47.598395 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:32:47.598469 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 07:32:47.599272 (+ 803us) server_negotiation.cc:299] Negotiation successful
0925 07:32:47.599378 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 07:32:47.601984 22865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fc18dd26bc34a0b8eac8010e3a9f734: Bootstrap starting.
I20260925 07:32:47.602751 22865 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0fc18dd26bc34a0b8eac8010e3a9f734: Neither blocks nor log segments found. Creating new log.
I20260925 07:32:47.603075 22865 log.cc:824] T 00000000000000000000000000000000 P 0fc18dd26bc34a0b8eac8010e3a9f734: Log is configured to *not* fsync() on all Append() calls
W20260925 07:32:47.604133 23041 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 024b1bbe80a4470b8090af8bc0ec3b59: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 07:32:47.604235 23041 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 07:32:47.604664 22944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:47.595914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42054 (local address 127.9.66.61:37987)
0925 07:32:47.595952 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 07:32:47.595954 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:32:47.596070 (+ 116us) server_negotiation.cc:407] Connection header received
0925 07:32:47.596163 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:32:47.596163 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:32:47.596173 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:32:47.596217 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:32:47.596974 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:32:47.597743 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:32:47.598731 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:32:47.599001 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:32:47.599057 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:32:47.600935 (+ 1878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:32:47.600937 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:32:47.600937 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:32:47.600944 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:32:47.603321 (+ 2377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:32:47.603702 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:32:47.603703 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:32:47.603703 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:32:47.603756 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:32:47.603952 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:32:47.603953 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:32:47.603953 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:32:47.604069 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:32:47.604176 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 07:32:47.604480 (+ 304us) server_negotiation.cc:299] Negotiation successful
0925 07:32:47.604554 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 07:32:47.604885 22865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fc18dd26bc34a0b8eac8010e3a9f734: No bootstrap required, opened a new log
I20260925 07:32:47.599483 23030 sys_catalog.cc:455] T 00000000000000000000000000000000 P 024b1bbe80a4470b8090af8bc0ec3b59 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fc18dd26bc34a0b8eac8010e3a9f734" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 39073 } } peers { permanent_uuid: "61cc89ef8d0f4893a84cd180cd92c209" member_type: VOTER last_known_addr { host: "127.9.66.61" port: 37987 } } peers { permanent_uuid: "024b1bbe80a4470b8090af8bc0ec3b59" member_type: VOTER last_known_addr { host: "127.9.66.60" port: 34971 } } }
I20260925 07:32:47.605628 23030 sys_catalog.cc:458] T 00000000000000000000000000000000 P 024b1bbe80a4470b8090af8bc0ec3b59 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 07:32:47.603358 22947 master_runner.cc:186] Error getting master registration for 127.9.66.62:39073: OK, instance_id { permanent_uuid: "0fc18dd26bc34a0b8eac8010e3a9f734" instance_seqno: 1790321567246476 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 07:32:47.605890 22947 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 07:32:47.606635 23028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:47.596401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33118 (local address 127.9.66.60:34971)
0925 07:32:47.597030 (+ 629us) server_negotiation.cc:206] Beginning negotiation
0925 07:32:47.597032 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:32:47.597118 (+ 86us) server_negotiation.cc:407] Connection header received
0925 07:32:47.597199 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:32:47.597200 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:32:47.597208 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:32:47.597246 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:32:47.597888 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:32:47.598639 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:32:47.599822 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:32:47.600153 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:32:47.600202 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:32:47.601688 (+ 1486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:32:47.601690 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:32:47.601690 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:32:47.601696 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:32:47.603409 (+ 1713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:32:47.604272 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:32:47.604273 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:32:47.604274 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:32:47.604359 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:32:47.604617 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:32:47.604618 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:32:47.604618 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:32:47.604734 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:32:47.604838 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 07:32:47.606522 (+ 1684us) server_negotiation.cc:299] Negotiation successful
0925 07:32:47.606573 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":609,"thread_start_us":70,"threads_started":1}
I20260925 07:32:47.607506 22865 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0fc18dd26bc34a0b8eac8010e3a9f734 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fc18dd26bc34a0b8eac8010e3a9f734" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 39073 } } peers { permanent_uuid: "61cc89ef8d0f4893a84cd180cd92c209" member_type: VOTER last_known_addr { host: "127.9.66.61" port: 37987 } } peers { permanent_uuid: "024b1bbe80a4470b8090af8bc0ec3b59" member_type: VOTER last_known_addr { host: "127.9.66.60" port: 34971 } }
I20260925 07:32:47.607674 22868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:47.595266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39046 (local address 127.9.66.62:39073)
0925 07:32:47.595306 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 07:32:47.595308 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:32:47.595615 (+ 307us) server_negotiation.cc:407] Connection header received
0925 07:32:47.595718 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:32:47.595719 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:32:47.595729 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:32:47.595763 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:32:47.600200 (+ 4437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:32:47.600833 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:32:47.601894 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:32:47.602144 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:32:47.602189 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:32:47.604895 (+ 2706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:32:47.604896 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:32:47.604897 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:32:47.604903 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:32:47.606435 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:32:47.606832 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:32:47.606833 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:32:47.606833 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:32:47.606903 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:32:47.607144 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:32:47.607145 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:32:47.607145 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:32:47.607292 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:32:47.607392 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 07:32:47.607538 (+ 146us) server_negotiation.cc:299] Negotiation successful
0925 07:32:47.607594 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 07:32:47.607653 22865 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0fc18dd26bc34a0b8eac8010e3a9f734 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:32:47.607823 22865 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0fc18dd26bc34a0b8eac8010e3a9f734 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fc18dd26bc34a0b8eac8010e3a9f734, State: Initialized, Role: FOLLOWER
I20260925 07:32:47.607947 22865 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0fc18dd26bc34a0b8eac8010e3a9f734 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fc18dd26bc34a0b8eac8010e3a9f734" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 39073 } } peers { permanent_uuid: "61cc89ef8d0f4893a84cd180cd92c209" member_type: VOTER last_known_addr { host: "127.9.66.61" port: 37987 } } peers { permanent_uuid: "024b1bbe80a4470b8090af8bc0ec3b59" member_type: VOTER last_known_addr { host: "127.9.66.60" port: 34971 } }
I20260925 07:32:47.608233 23044 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fc18dd26bc34a0b8eac8010e3a9f734 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fc18dd26bc34a0b8eac8010e3a9f734" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 39073 } } peers { permanent_uuid: "61cc89ef8d0f4893a84cd180cd92c209" member_type: VOTER last_known_addr { host: "127.9.66.61" port: 37987 } } peers { permanent_uuid: "024b1bbe80a4470b8090af8bc0ec3b59" member_type: VOTER last_known_addr { host: "127.9.66.60" port: 34971 } } }
I20260925 07:32:47.608234 22865 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0fc18dd26bc34a0b8eac8010e3a9f734 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:32:47.608346 23044 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fc18dd26bc34a0b8eac8010e3a9f734 [sys.catalog]: This master's current role is: FOLLOWER
W20260925 07:32:47.610096 23055 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0fc18dd26bc34a0b8eac8010e3a9f734: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 07:32:47.610188 23055 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 07:32:47.611001 22797 master_runner.cc:186] Error getting master registration for 127.9.66.61:37987: OK, instance_id { permanent_uuid: "61cc89ef8d0f4893a84cd180cd92c209" instance_seqno: 1790321567399424 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 07:32:47.612977 22797 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 07:32:47.637545 22940 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.66.60" port: 34971 } attempt: 3
Sep 25 07:32:47 dist-test-slave-57fn krb5kdc[22784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321567, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.61@KRBTEST.COM for kudu/127.9.66.60@KRBTEST.COM
I20260925 07:32:47.644601 23028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:47.637903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33124 (local address 127.9.66.60:34971)
0925 07:32:47.637941 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 07:32:47.637943 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:32:47.637990 (+ 47us) server_negotiation.cc:407] Connection header received
0925 07:32:47.638051 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:32:47.638052 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:32:47.638065 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:32:47.638149 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:32:47.638449 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:32:47.638938 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:32:47.639898 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:32:47.640122 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:32:47.640199 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:32:47.642051 (+ 1852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:32:47.642052 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:32:47.642053 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:32:47.642060 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:32:47.643495 (+ 1435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:32:47.643870 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:32:47.643872 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:32:47.643872 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:32:47.643928 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:32:47.644118 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:32:47.644119 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:32:47.644120 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:32:47.644225 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:32:47.644299 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 07:32:47.644438 (+ 139us) server_negotiation.cc:299] Negotiation successful
0925 07:32:47.644508 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 07:32:47.644600 22942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:47.637867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.60:34971 (local address 127.0.0.1:33124)
0925 07:32:47.637929 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0925 07:32:47.637941 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 07:32:47.637989 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:32:47.638154 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:32:47.638155 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:32:47.638322 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:32:47.638363 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:32:47.638364 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:32:47.639028 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:32:47.639029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:32:47.639799 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:32:47.639801 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:32:47.640195 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:32:47.640196 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:32:47.640251 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:32:47.640714 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:32:47.640720 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:32:47.641956 (+ 1236us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:32:47.643606 (+ 1650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:32:47.643607 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:32:47.643607 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:32:47.643786 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:32:47.643993 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:32:47.643993 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:32:47.643994 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:32:47.644055 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:32:47.644290 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:32:47.644291 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:32:47.644332 (+ 41us) client_negotiation.cc:769] Sending connection context
0925 07:32:47.644414 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 07:32:47.644506 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260925 07:32:47.646230 22940 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61cc89ef8d0f4893a84cd180cd92c209: Bootstrap starting.
I20260925 07:32:47.646863 22940 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 61cc89ef8d0f4893a84cd180cd92c209: Neither blocks nor log segments found. Creating new log.
I20260925 07:32:47.647243 22940 log.cc:824] T 00000000000000000000000000000000 P 61cc89ef8d0f4893a84cd180cd92c209: Log is configured to *not* fsync() on all Append() calls
I20260925 07:32:47.648056 22940 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61cc89ef8d0f4893a84cd180cd92c209: No bootstrap required, opened a new log
I20260925 07:32:47.650408 22940 raft_consensus.cc:348] T 00000000000000000000000000000000 P 61cc89ef8d0f4893a84cd180cd92c209 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fc18dd26bc34a0b8eac8010e3a9f734" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 39073 } } peers { permanent_uuid: "61cc89ef8d0f4893a84cd180cd92c209" member_type: VOTER last_known_addr { host: "127.9.66.61" port: 37987 } } peers { permanent_uuid: "024b1bbe80a4470b8090af8bc0ec3b59" member_type: VOTER last_known_addr { host: "127.9.66.60" port: 34971 } }
I20260925 07:32:47.650563 22940 raft_consensus.cc:374] T 00000000000000000000000000000000 P 61cc89ef8d0f4893a84cd180cd92c209 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:32:47.650635 22940 raft_consensus.cc:729] T 00000000000000000000000000000000 P 61cc89ef8d0f4893a84cd180cd92c209 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61cc89ef8d0f4893a84cd180cd92c209, State: Initialized, Role: FOLLOWER
I20260925 07:32:47.650789 22940 consensus_queue.cc:260] T 00000000000000000000000000000000 P 61cc89ef8d0f4893a84cd180cd92c209 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fc18dd26bc34a0b8eac8010e3a9f734" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 39073 } } peers { permanent_uuid: "61cc89ef8d0f4893a84cd180cd92c209" member_type: VOTER last_known_addr { host: "127.9.66.61" port: 37987 } } peers { permanent_uuid: "024b1bbe80a4470b8090af8bc0ec3b59" member_type: VOTER last_known_addr { host: "127.9.66.60" port: 34971 } }
I20260925 07:32:47.651060 23058 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61cc89ef8d0f4893a84cd180cd92c209 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fc18dd26bc34a0b8eac8010e3a9f734" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 39073 } } peers { permanent_uuid: "61cc89ef8d0f4893a84cd180cd92c209" member_type: VOTER last_known_addr { host: "127.9.66.61" port: 37987 } } peers { permanent_uuid: "024b1bbe80a4470b8090af8bc0ec3b59" member_type: VOTER last_known_addr { host: "127.9.66.60" port: 34971 } } }
I20260925 07:32:47.651132 22940 sys_catalog.cc:564] T 00000000000000000000000000000000 P 61cc89ef8d0f4893a84cd180cd92c209 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:32:47.651194 23058 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61cc89ef8d0f4893a84cd180cd92c209 [sys.catalog]: This master's current role is: FOLLOWER
W20260925 07:32:47.652928 23069 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 61cc89ef8d0f4893a84cd180cd92c209: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 07:32:47.653034 23069 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 07:32:47.655120 22872 master_runner.cc:263] No leader master found from master 61cc89ef8d0f4893a84cd180cd92c209
I20260925 07:32:47.655169 22872 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260925 07:32:47.744570 22818 tablet.cc:2369] T 00000000000000000000000000000000 P 0fc18dd26bc34a0b8eac8010e3a9f734: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 07:32:47.760190 23030 raft_consensus.cc:482] T 00000000000000000000000000000000 P 024b1bbe80a4470b8090af8bc0ec3b59 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 07:32:47.760320 23030 raft_consensus.cc:504] T 00000000000000000000000000000000 P 024b1bbe80a4470b8090af8bc0ec3b59 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fc18dd26bc34a0b8eac8010e3a9f734" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 39073 } } peers { permanent_uuid: "61cc89ef8d0f4893a84cd180cd92c209" member_type: VOTER last_known_addr { host: "127.9.66.61" port: 37987 } } peers { permanent_uuid: "024b1bbe80a4470b8090af8bc0ec3b59" member_type: VOTER last_known_addr { host: "127.9.66.60" port: 34971 } }
I20260925 07:32:47.760572 23030 leader_election.cc:288] T 00000000000000000000000000000000 P 024b1bbe80a4470b8090af8bc0ec3b59 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fc18dd26bc34a0b8eac8010e3a9f734 (127.9.66.62:39073), 61cc89ef8d0f4893a84cd180cd92c209 (127.9.66.61:37987)
I20260925 07:32:47.761126 22839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "024b1bbe80a4470b8090af8bc0ec3b59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fc18dd26bc34a0b8eac8010e3a9f734" is_pre_election: true
I20260925 07:32:47.761277 22839 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0fc18dd26bc34a0b8eac8010e3a9f734 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 024b1bbe80a4470b8090af8bc0ec3b59 in term 0.
I20260925 07:32:47.761855 22967 leader_election.cc:302] T 00000000000000000000000000000000 P 024b1bbe80a4470b8090af8bc0ec3b59 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 024b1bbe80a4470b8090af8bc0ec3b59, 0fc18dd26bc34a0b8eac8010e3a9f734; no voters:
I20260925 07:32:47.762563 22914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "024b1bbe80a4470b8090af8bc0ec3b59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61cc89ef8d0f4893a84cd180cd92c209" is_pre_election: true
I20260925 07:32:47.762712 22914 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 61cc89ef8d0f4893a84cd180cd92c209 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 024b1bbe80a4470b8090af8bc0ec3b59 in term 0.
I20260925 07:32:47.762894 23030 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 024b1bbe80a4470b8090af8bc0ec3b59 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 07:32:47.762960 23030 raft_consensus.cc:482] T 00000000000000000000000000000000 P 024b1bbe80a4470b8090af8bc0ec3b59 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 07:32:47.762984 23030 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 024b1bbe80a4470b8090af8bc0ec3b59 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:32:47.763921 23030 raft_consensus.cc:504] T 00000000000000000000000000000000 P 024b1bbe80a4470b8090af8bc0ec3b59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fc18dd26bc34a0b8eac8010e3a9f734" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 39073 } } peers { permanent_uuid: "61cc89ef8d0f4893a84cd180cd92c209" member_type: VOTER last_known_addr { host: "127.9.66.61" port: 37987 } } peers { permanent_uuid: "024b1bbe80a4470b8090af8bc0ec3b59" member_type: VOTER last_known_addr { host: "127.9.66.60" port: 34971 } }
I20260925 07:32:47.764045 23030 leader_election.cc:288] T 00000000000000000000000000000000 P 024b1bbe80a4470b8090af8bc0ec3b59 [CANDIDATE]: Term 1 election: Requested vote from peers 0fc18dd26bc34a0b8eac8010e3a9f734 (127.9.66.62:39073), 61cc89ef8d0f4893a84cd180cd92c209 (127.9.66.61:37987)
I20260925 07:32:47.764436 22914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "024b1bbe80a4470b8090af8bc0ec3b59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61cc89ef8d0f4893a84cd180cd92c209"
I20260925 07:32:47.764536 22914 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 61cc89ef8d0f4893a84cd180cd92c209 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:32:47.765409 22914 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 61cc89ef8d0f4893a84cd180cd92c209 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 024b1bbe80a4470b8090af8bc0ec3b59 in term 1.
I20260925 07:32:47.765668 22964 leader_election.cc:302] T 00000000000000000000000000000000 P 024b1bbe80a4470b8090af8bc0ec3b59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 024b1bbe80a4470b8090af8bc0ec3b59, 61cc89ef8d0f4893a84cd180cd92c209; no voters:
I20260925 07:32:47.765915 22839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "024b1bbe80a4470b8090af8bc0ec3b59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fc18dd26bc34a0b8eac8010e3a9f734"
I20260925 07:32:47.766002 22839 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0fc18dd26bc34a0b8eac8010e3a9f734 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:32:47.766834 22839 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0fc18dd26bc34a0b8eac8010e3a9f734 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 024b1bbe80a4470b8090af8bc0ec3b59 in term 1.
I20260925 07:32:47.768023 23030 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 024b1bbe80a4470b8090af8bc0ec3b59 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:32:47.768163 23030 raft_consensus.cc:686] T 00000000000000000000000000000000 P 024b1bbe80a4470b8090af8bc0ec3b59 [term 1 LEADER]: Becoming Leader. State: Replica: 024b1bbe80a4470b8090af8bc0ec3b59, State: Running, Role: LEADER
I20260925 07:32:47.768236 23030 consensus_queue.cc:237] T 00000000000000000000000000000000 P 024b1bbe80a4470b8090af8bc0ec3b59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fc18dd26bc34a0b8eac8010e3a9f734" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 39073 } } peers { permanent_uuid: "61cc89ef8d0f4893a84cd180cd92c209" member_type: VOTER last_known_addr { host: "127.9.66.61" port: 37987 } } peers { permanent_uuid: "024b1bbe80a4470b8090af8bc0ec3b59" member_type: VOTER last_known_addr { host: "127.9.66.60" port: 34971 } }
I20260925 07:32:47.768580 23030 sys_catalog.cc:455] T 00000000000000000000000000000000 P 024b1bbe80a4470b8090af8bc0ec3b59 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 024b1bbe80a4470b8090af8bc0ec3b59. Latest consensus state: current_term: 1 leader_uuid: "024b1bbe80a4470b8090af8bc0ec3b59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fc18dd26bc34a0b8eac8010e3a9f734" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 39073 } } peers { permanent_uuid: "61cc89ef8d0f4893a84cd180cd92c209" member_type: VOTER last_known_addr { host: "127.9.66.61" port: 37987 } } peers { permanent_uuid: "024b1bbe80a4470b8090af8bc0ec3b59" member_type: VOTER last_known_addr { host: "127.9.66.60" port: 34971 } } }
I20260925 07:32:47.768635 23030 sys_catalog.cc:458] T 00000000000000000000000000000000 P 024b1bbe80a4470b8090af8bc0ec3b59 [sys.catalog]: This master's current role is: LEADER
I20260925 07:32:47.768930 23073 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 07:32:47.769135 23073 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 07:32:47.771051 22914 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 61cc89ef8d0f4893a84cd180cd92c209 [term 1 FOLLOWER]: Refusing update from remote peer 024b1bbe80a4470b8090af8bc0ec3b59: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:32:47.771286 22839 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0fc18dd26bc34a0b8eac8010e3a9f734 [term 1 FOLLOWER]: Refusing update from remote peer 024b1bbe80a4470b8090af8bc0ec3b59: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:32:47.771764 23030 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 024b1bbe80a4470b8090af8bc0ec3b59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61cc89ef8d0f4893a84cd180cd92c209" member_type: VOTER last_known_addr { host: "127.9.66.61" port: 37987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:32:47.771889 23030 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 024b1bbe80a4470b8090af8bc0ec3b59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fc18dd26bc34a0b8eac8010e3a9f734" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 39073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:32:47.773115 23058 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61cc89ef8d0f4893a84cd180cd92c209 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 024b1bbe80a4470b8090af8bc0ec3b59. Latest consensus state: current_term: 1 leader_uuid: "024b1bbe80a4470b8090af8bc0ec3b59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fc18dd26bc34a0b8eac8010e3a9f734" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 39073 } } peers { permanent_uuid: "61cc89ef8d0f4893a84cd180cd92c209" member_type: VOTER last_known_addr { host: "127.9.66.61" port: 37987 } } peers { permanent_uuid: "024b1bbe80a4470b8090af8bc0ec3b59" member_type: VOTER last_known_addr { host: "127.9.66.60" port: 34971 } } }
I20260925 07:32:47.774293 23058 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61cc89ef8d0f4893a84cd180cd92c209 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 07:32:47.775285 23044 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fc18dd26bc34a0b8eac8010e3a9f734 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 024b1bbe80a4470b8090af8bc0ec3b59. Latest consensus state: current_term: 1 leader_uuid: "024b1bbe80a4470b8090af8bc0ec3b59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fc18dd26bc34a0b8eac8010e3a9f734" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 39073 } } peers { permanent_uuid: "61cc89ef8d0f4893a84cd180cd92c209" member_type: VOTER last_known_addr { host: "127.9.66.61" port: 37987 } } peers { permanent_uuid: "024b1bbe80a4470b8090af8bc0ec3b59" member_type: VOTER last_known_addr { host: "127.9.66.60" port: 34971 } } }
I20260925 07:32:47.775367 23044 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fc18dd26bc34a0b8eac8010e3a9f734 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 07:32:47.776953 23030 sys_catalog.cc:455] T 00000000000000000000000000000000 P 024b1bbe80a4470b8090af8bc0ec3b59 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "024b1bbe80a4470b8090af8bc0ec3b59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fc18dd26bc34a0b8eac8010e3a9f734" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 39073 } } peers { permanent_uuid: "61cc89ef8d0f4893a84cd180cd92c209" member_type: VOTER last_known_addr { host: "127.9.66.61" port: 37987 } } peers { permanent_uuid: "024b1bbe80a4470b8090af8bc0ec3b59" member_type: VOTER last_known_addr { host: "127.9.66.60" port: 34971 } } }
I20260925 07:32:47.777026 23030 sys_catalog.cc:458] T 00000000000000000000000000000000 P 024b1bbe80a4470b8090af8bc0ec3b59 [sys.catalog]: This master's current role is: LEADER
I20260925 07:32:47.777403 23044 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fc18dd26bc34a0b8eac8010e3a9f734 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "024b1bbe80a4470b8090af8bc0ec3b59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fc18dd26bc34a0b8eac8010e3a9f734" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 39073 } } peers { permanent_uuid: "61cc89ef8d0f4893a84cd180cd92c209" member_type: VOTER last_known_addr { host: "127.9.66.61" port: 37987 } } peers { permanent_uuid: "024b1bbe80a4470b8090af8bc0ec3b59" member_type: VOTER last_known_addr { host: "127.9.66.60" port: 34971 } } }
I20260925 07:32:47.777475 23044 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fc18dd26bc34a0b8eac8010e3a9f734 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 07:32:47.777830 23074 mvcc.cc:205] Tried to move back new op lower bound from 7333157141587755008 to 7333157141579210752. Current Snapshot: MvccSnapshot[applied={T|T < 7333157141587755008}]
I20260925 07:32:47.778287 23030 sys_catalog.cc:455] T 00000000000000000000000000000000 P 024b1bbe80a4470b8090af8bc0ec3b59 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "024b1bbe80a4470b8090af8bc0ec3b59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fc18dd26bc34a0b8eac8010e3a9f734" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 39073 } } peers { permanent_uuid: "61cc89ef8d0f4893a84cd180cd92c209" member_type: VOTER last_known_addr { host: "127.9.66.61" port: 37987 } } peers { permanent_uuid: "024b1bbe80a4470b8090af8bc0ec3b59" member_type: VOTER last_known_addr { host: "127.9.66.60" port: 34971 } } }
I20260925 07:32:47.778357 23030 sys_catalog.cc:458] T 00000000000000000000000000000000 P 024b1bbe80a4470b8090af8bc0ec3b59 [sys.catalog]: This master's current role is: LEADER
I20260925 07:32:47.778721 23058 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61cc89ef8d0f4893a84cd180cd92c209 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "024b1bbe80a4470b8090af8bc0ec3b59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fc18dd26bc34a0b8eac8010e3a9f734" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 39073 } } peers { permanent_uuid: "61cc89ef8d0f4893a84cd180cd92c209" member_type: VOTER last_known_addr { host: "127.9.66.61" port: 37987 } } peers { permanent_uuid: "024b1bbe80a4470b8090af8bc0ec3b59" member_type: VOTER last_known_addr { host: "127.9.66.60" port: 34971 } } }
I20260925 07:32:47.778795 23058 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61cc89ef8d0f4893a84cd180cd92c209 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 07:32:47.779549 23075 mvcc.cc:205] Tried to move back new op lower bound from 7333157141587755008 to 7333157141579210752. Current Snapshot: MvccSnapshot[applied={T|T < 7333157141587755008}]
I20260925 07:32:47.779983 23077 mvcc.cc:205] Tried to move back new op lower bound from 7333157141587755008 to 7333157141579210752. Current Snapshot: MvccSnapshot[applied={T|T < 7333157141587755008}]
I20260925 07:32:47.780575 23073 catalog_manager.cc:1347] Generated new cluster ID: 4d3e096f8f2445778239bd2ba52bed98
I20260925 07:32:47.780618 23073 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 07:32:47.810431 23073 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 07:32:47.811067 23073 catalog_manager.cc:1504] Loading token signing keys...
I20260925 07:32:47.813766 23073 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 024b1bbe80a4470b8090af8bc0ec3b59: Generated new TSK 0
I20260925 07:32:47.813967 23073 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 07:32:48.610661 23055 catalog_manager.cc:1259] Loaded cluster ID: 4d3e096f8f2445778239bd2ba52bed98
I20260925 07:32:48.610737 23055 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0fc18dd26bc34a0b8eac8010e3a9f734: loading cluster ID for follower catalog manager: success
I20260925 07:32:48.611552 23055 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0fc18dd26bc34a0b8eac8010e3a9f734: acquiring CA information for follower catalog manager: success
I20260925 07:32:48.611869 23055 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0fc18dd26bc34a0b8eac8010e3a9f734: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 07:32:48.653469 23069 catalog_manager.cc:1259] Loaded cluster ID: 4d3e096f8f2445778239bd2ba52bed98
I20260925 07:32:48.653564 23069 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 61cc89ef8d0f4893a84cd180cd92c209: loading cluster ID for follower catalog manager: success
I20260925 07:32:48.654366 23069 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 61cc89ef8d0f4893a84cd180cd92c209: acquiring CA information for follower catalog manager: success
I20260925 07:32:48.654675 23069 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 61cc89ef8d0f4893a84cd180cd92c209: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 07:32:48.869835 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 22797
I20260925 07:32:48.875200 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 22872
I20260925 07:32:48.880055 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 22947
2026-09-25T07:32:48Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2893 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790321392278483-9480-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 07:32:48 dist-test-slave-57fn krb5kdc[23089](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 07:32:48 dist-test-slave-57fn krb5kdc[23089](info): set up 2 sockets
Sep 25 07:32:48 dist-test-slave-57fn krb5kdc[23089](info): commencing operation
krb5kdc: starting...
W20260925 07:32:49.902042 9480 mini_kdc.cc:121] Time spent starting KDC: real 1.003s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790321392278483-9480-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790321392278483-9480-0/krb5kdc/test-user.keytab.
Sep 25 07:32:49 dist-test-slave-57fn krb5kdc[23089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321569, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T07:32:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T07:32:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.66.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.66.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.66.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab.
Entry for principal kudu/127.9.66.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab.
I20260925 07:32:50.013104 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790321392278483-9480-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790321392278483-9480-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790321392278483-9480-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.66.62:40941
--webserver_interface=127.9.66.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab
--principal=kudu/127.9.66.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:41595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.66.62:40941
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:32:50.092175 23102 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:32:50.092397 23102 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:32:50.092433 23102 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:32:50.093577 23102 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:32:50.093640 23102 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:32:50.093664 23102 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:32:50.093686 23102 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:32:50.093714 23102 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:32:50.095125 23102 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:41595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790321392278483-9480-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790321392278483-9480-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.66.62:40941
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab
--principal=kudu/127.9.66.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.66.62:40941
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.66.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23102
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790321392278483-9480-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:32:50.095394 23102 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:32:50.095610 23102 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:32:50.100342 23110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:32:50.100309 23107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:32:50.100327 23102 server_base.cc:1034] running on GCE node
W20260925 07:32:50.100315 23108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:32:50.100832 23102 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:32:50.101115 23102 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:32:50.102257 23102 hybrid_clock.cc:648] HybridClock initialized: now 1790321570102249 us; error 35 us; skew 500 ppm
Sep 25 07:32:50 dist-test-slave-57fn krb5kdc[23089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321570, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:32:50.116235 23102 init.cc:377] Logged in from keytab as kudu/127.9.66.62@KRBTEST.COM (short username kudu)
I20260925 07:32:50.116657 23102 webserver.cc:460] Webserver started at http://127.9.66.62:33473/ using document root <none> and password file <none>
I20260925 07:32:50.116886 23102 fs_manager.cc:359] Metadata directory not provided
I20260925 07:32:50.116954 23102 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:32:50.117115 23102 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:32:50.118256 23102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790321392278483-9480-0/minicluster-data/master-0/data/instance:
uuid: "6f955641f78649a490f1cc4ded2a8296"
format_stamp: "Formatted at 2026-09-25 07:32:50 on dist-test-slave-57fn"
server_key: "614272fa064b174ce1ba0558d64430dd"
server_key_iv: "426bd1d2396fe4702b0a89b045dd9cae"
server_key_version: "encryptionkey@0"
I20260925 07:32:50.118608 23102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790321392278483-9480-0/minicluster-data/master-0/wal/instance:
uuid: "6f955641f78649a490f1cc4ded2a8296"
format_stamp: "Formatted at 2026-09-25 07:32:50 on dist-test-slave-57fn"
server_key: "614272fa064b174ce1ba0558d64430dd"
server_key_iv: "426bd1d2396fe4702b0a89b045dd9cae"
server_key_version: "encryptionkey@0"
I20260925 07:32:50.119885 23102 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 07:32:50.120592 23117 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:32:50.120770 23102 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 07:32:50.120867 23102 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790321392278483-9480-0/minicluster-data/master-0/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790321392278483-9480-0/minicluster-data/master-0/wal
uuid: "6f955641f78649a490f1cc4ded2a8296"
format_stamp: "Formatted at 2026-09-25 07:32:50 on dist-test-slave-57fn"
server_key: "614272fa064b174ce1ba0558d64430dd"
server_key_iv: "426bd1d2396fe4702b0a89b045dd9cae"
server_key_version: "encryptionkey@0"
I20260925 07:32:50.120944 23102 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790321392278483-9480-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790321392278483-9480-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790321392278483-9480-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:32:50.133035 23102 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:32:50.134958 23102 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:32:50.135133 23102 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:32:50.139341 23102 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.62:40941
I20260925 07:32:50.139426 23169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.62:40941 every 8 connection(s)
I20260925 07:32:50.139760 23102 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
I20260925 07:32:50.140563 23170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:32:50.142781 23170 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f955641f78649a490f1cc4ded2a8296: Bootstrap starting.
I20260925 07:32:50.143288 23170 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6f955641f78649a490f1cc4ded2a8296: Neither blocks nor log segments found. Creating new log.
I20260925 07:32:50.143507 23170 log.cc:824] T 00000000000000000000000000000000 P 6f955641f78649a490f1cc4ded2a8296: Log is configured to *not* fsync() on all Append() calls
I20260925 07:32:50.144099 23170 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f955641f78649a490f1cc4ded2a8296: No bootstrap required, opened a new log
I20260925 07:32:50.145646 23170 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6f955641f78649a490f1cc4ded2a8296 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f955641f78649a490f1cc4ded2a8296" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 40941 } }
I20260925 07:32:50.145776 23170 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6f955641f78649a490f1cc4ded2a8296 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:32:50.145797 23170 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6f955641f78649a490f1cc4ded2a8296 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f955641f78649a490f1cc4ded2a8296, State: Initialized, Role: FOLLOWER
I20260925 07:32:50.145931 23170 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f955641f78649a490f1cc4ded2a8296 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f955641f78649a490f1cc4ded2a8296" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 40941 } }
I20260925 07:32:50.146009 23170 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6f955641f78649a490f1cc4ded2a8296 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 07:32:50.146032 23170 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6f955641f78649a490f1cc4ded2a8296 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 07:32:50.146059 23170 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6f955641f78649a490f1cc4ded2a8296 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:32:50.146502 23170 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6f955641f78649a490f1cc4ded2a8296 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f955641f78649a490f1cc4ded2a8296" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 40941 } }
I20260925 07:32:50.146580 23170 leader_election.cc:302] T 00000000000000000000000000000000 P 6f955641f78649a490f1cc4ded2a8296 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6f955641f78649a490f1cc4ded2a8296; no voters:
I20260925 07:32:50.146697 23170 leader_election.cc:288] T 00000000000000000000000000000000 P 6f955641f78649a490f1cc4ded2a8296 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 07:32:50.146752 23173 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6f955641f78649a490f1cc4ded2a8296 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:32:50.146950 23170 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f955641f78649a490f1cc4ded2a8296 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:32:50.146955 23173 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6f955641f78649a490f1cc4ded2a8296 [term 1 LEADER]: Becoming Leader. State: Replica: 6f955641f78649a490f1cc4ded2a8296, State: Running, Role: LEADER
I20260925 07:32:50.147145 23173 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6f955641f78649a490f1cc4ded2a8296 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f955641f78649a490f1cc4ded2a8296" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 40941 } }
I20260925 07:32:50.147509 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 23102
I20260925 07:32:50.147614 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790321392278483-9480-0/minicluster-data/master-0/wal/instance
I20260925 07:32:50.147616 23174 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f955641f78649a490f1cc4ded2a8296 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6f955641f78649a490f1cc4ded2a8296" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f955641f78649a490f1cc4ded2a8296" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 40941 } } }
I20260925 07:32:50.147694 23174 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f955641f78649a490f1cc4ded2a8296 [sys.catalog]: This master's current role is: LEADER
I20260925 07:32:50.147845 9480 external_mini_cluster.cc:1428] Setting key 4b6858d02c613d66cb902f72fc6e1af7
I20260925 07:32:50.148154 23175 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f955641f78649a490f1cc4ded2a8296 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f955641f78649a490f1cc4ded2a8296. Latest consensus state: current_term: 1 leader_uuid: "6f955641f78649a490f1cc4ded2a8296" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f955641f78649a490f1cc4ded2a8296" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 40941 } } }
I20260925 07:32:50.148218 23175 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f955641f78649a490f1cc4ded2a8296 [sys.catalog]: This master's current role is: LEADER
I20260925 07:32:50.148352 23182 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 07:32:50.149292 23182 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 07:32:50.150791 23182 catalog_manager.cc:1347] Generated new cluster ID: 7348ac8c221e49b5808d674499e2a944
I20260925 07:32:50.150846 23182 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 07:32:50 dist-test-slave-57fn krb5kdc[23089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321569, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.66.62@KRBTEST.COM
I20260925 07:32:50.158607 23192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:50.148538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35944 (local address 127.9.66.62:40941)
0925 07:32:50.149314 (+ 776us) server_negotiation.cc:206] Beginning negotiation
0925 07:32:50.149323 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 07:32:50.149355 (+ 32us) server_negotiation.cc:407] Connection header received
0925 07:32:50.149463 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:32:50.149468 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:32:50.149562 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:32:50.149714 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:32:50.150462 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:32:50.151494 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:32:50.152216 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:32:50.152624 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:32:50.152732 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:32:50.155061 (+ 2329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:32:50.155065 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:32:50.155065 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:32:50.155075 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:32:50.157112 (+ 2037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:32:50.157494 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:32:50.157497 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:32:50.157498 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:32:50.157616 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:32:50.157821 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:32:50.157822 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:32:50.157822 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:32:50.158063 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:32:50.158234 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0925 07:32:50.158309 (+ 75us) server_negotiation.cc:299] Negotiation successful
0925 07:32:50.158447 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":727,"thread_start_us":74,"threads_started":1}
I20260925 07:32:50.159435 9480 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 07:32:50.165025 23182 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 07:32:50.165724 23182 catalog_manager.cc:1504] Loading token signing keys...
I20260925 07:32:50.170447 23182 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6f955641f78649a490f1cc4ded2a8296: Generated new TSK 0
I20260925 07:32:50.170593 23182 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 07:32:50.213294 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 23102
2026-09-25T07:32:50Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1331 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 07:32:50 dist-test-slave-57fn krb5kdc[23198](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 07:32:50 dist-test-slave-57fn krb5kdc[23198](info): set up 2 sockets
Sep 25 07:32:50 dist-test-slave-57fn krb5kdc[23198](info): commencing operation
krb5kdc: starting...
W20260925 07:32:51.225589 9480 mini_kdc.cc:121] Time spent starting KDC: real 0.994s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/krb5kdc/test-user.keytab.
Sep 25 07:32:51 dist-test-slave-57fn krb5kdc[23198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321571, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T07:32:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T07:32:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.66.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.66.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.66.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab.
Entry for principal kudu/127.9.66.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab.
I20260925 07:32:51.340170 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.66.62:46577
--webserver_interface=127.9.66.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab
--principal=kudu/127.9.66.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:35859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.66.62:46577
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:32:51.417039 23211 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:32:51.417271 23211 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:32:51.417306 23211 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:32:51.418434 23211 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:32:51.418514 23211 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:32:51.418537 23211 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:32:51.418565 23211 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:32:51.418593 23211 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:32:51.420234 23211 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:35859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.66.62:46577
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab
--principal=kudu/127.9.66.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.66.62:46577
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.66.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23211
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:32:51.420498 23211 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:32:51.420739 23211 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:32:51.426419 23217 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:32:51.426424 23216 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:32:51.426442 23219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:32:51.426699 23211 server_base.cc:1034] running on GCE node
I20260925 07:32:51.426951 23211 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:32:51.427281 23211 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:32:51.428434 23211 hybrid_clock.cc:648] HybridClock initialized: now 1790321571428418 us; error 30 us; skew 500 ppm
Sep 25 07:32:51 dist-test-slave-57fn krb5kdc[23198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321571, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:32:51.430992 23211 init.cc:377] Logged in from keytab as kudu/127.9.66.62@KRBTEST.COM (short username kudu)
I20260925 07:32:51.431336 23211 webserver.cc:460] Webserver started at http://127.9.66.62:35781/ using document root <none> and password file <none>
I20260925 07:32:51.431520 23211 fs_manager.cc:359] Metadata directory not provided
I20260925 07:32:51.431558 23211 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:32:51.431640 23211 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:32:51.432605 23211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/master-0/data/instance:
uuid: "265a353e5527411ca0c035891c95121e"
format_stamp: "Formatted at 2026-09-25 07:32:51 on dist-test-slave-57fn"
server_key: "4f72d3af8ddb6c5ae0dd1db167acd81a"
server_key_iv: "eacc4d534c2689883e186a27aea4d983"
server_key_version: "encryptionkey@0"
I20260925 07:32:51.432914 23211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/master-0/wal/instance:
uuid: "265a353e5527411ca0c035891c95121e"
format_stamp: "Formatted at 2026-09-25 07:32:51 on dist-test-slave-57fn"
server_key: "4f72d3af8ddb6c5ae0dd1db167acd81a"
server_key_iv: "eacc4d534c2689883e186a27aea4d983"
server_key_version: "encryptionkey@0"
I20260925 07:32:51.434207 23211 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 07:32:51.434870 23226 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:32:51.435055 23211 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 07:32:51.435140 23211 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/master-0/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/master-0/wal
uuid: "265a353e5527411ca0c035891c95121e"
format_stamp: "Formatted at 2026-09-25 07:32:51 on dist-test-slave-57fn"
server_key: "4f72d3af8ddb6c5ae0dd1db167acd81a"
server_key_iv: "eacc4d534c2689883e186a27aea4d983"
server_key_version: "encryptionkey@0"
I20260925 07:32:51.435207 23211 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:32:51.456261 23211 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:32:51.458256 23211 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:32:51.458428 23211 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:32:51.462316 23211 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.62:46577
I20260925 07:32:51.462366 23278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.62:46577 every 8 connection(s)
I20260925 07:32:51.462733 23211 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
I20260925 07:32:51.463637 23279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:32:51.464289 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 23211
I20260925 07:32:51.464389 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/master-0/wal/instance
I20260925 07:32:51.464563 9480 external_mini_cluster.cc:1428] Setting key 6558f985a7f14670caf7379b4d86f230
I20260925 07:32:51.466912 23279 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 265a353e5527411ca0c035891c95121e: Bootstrap starting.
I20260925 07:32:51.467489 23279 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 265a353e5527411ca0c035891c95121e: Neither blocks nor log segments found. Creating new log.
I20260925 07:32:51.467774 23279 log.cc:824] T 00000000000000000000000000000000 P 265a353e5527411ca0c035891c95121e: Log is configured to *not* fsync() on all Append() calls
I20260925 07:32:51.468497 23279 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 265a353e5527411ca0c035891c95121e: No bootstrap required, opened a new log
I20260925 07:32:51.470935 23279 raft_consensus.cc:348] T 00000000000000000000000000000000 P 265a353e5527411ca0c035891c95121e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "265a353e5527411ca0c035891c95121e" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 46577 } }
I20260925 07:32:51.471094 23279 raft_consensus.cc:374] T 00000000000000000000000000000000 P 265a353e5527411ca0c035891c95121e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:32:51.471118 23279 raft_consensus.cc:729] T 00000000000000000000000000000000 P 265a353e5527411ca0c035891c95121e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 265a353e5527411ca0c035891c95121e, State: Initialized, Role: FOLLOWER
I20260925 07:32:51.471249 23279 consensus_queue.cc:260] T 00000000000000000000000000000000 P 265a353e5527411ca0c035891c95121e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "265a353e5527411ca0c035891c95121e" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 46577 } }
I20260925 07:32:51.471331 23279 raft_consensus.cc:388] T 00000000000000000000000000000000 P 265a353e5527411ca0c035891c95121e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 07:32:51.471354 23279 raft_consensus.cc:482] T 00000000000000000000000000000000 P 265a353e5527411ca0c035891c95121e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 07:32:51.471421 23279 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 265a353e5527411ca0c035891c95121e [term 0 FOLLOWER]: Advancing to term 1
Sep 25 07:32:51 dist-test-slave-57fn krb5kdc[23198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321571, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.66.62@KRBTEST.COM
I20260925 07:32:51.471979 23279 raft_consensus.cc:504] T 00000000000000000000000000000000 P 265a353e5527411ca0c035891c95121e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "265a353e5527411ca0c035891c95121e" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 46577 } }
I20260925 07:32:51.472103 23279 leader_election.cc:302] T 00000000000000000000000000000000 P 265a353e5527411ca0c035891c95121e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 265a353e5527411ca0c035891c95121e; no voters:
I20260925 07:32:51.472263 23279 leader_election.cc:288] T 00000000000000000000000000000000 P 265a353e5527411ca0c035891c95121e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 07:32:51.472518 23279 sys_catalog.cc:564] T 00000000000000000000000000000000 P 265a353e5527411ca0c035891c95121e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:32:51.472620 23284 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 265a353e5527411ca0c035891c95121e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:32:51.472769 23284 raft_consensus.cc:686] T 00000000000000000000000000000000 P 265a353e5527411ca0c035891c95121e [term 1 LEADER]: Becoming Leader. State: Replica: 265a353e5527411ca0c035891c95121e, State: Running, Role: LEADER
I20260925 07:32:51.472921 23284 consensus_queue.cc:237] T 00000000000000000000000000000000 P 265a353e5527411ca0c035891c95121e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "265a353e5527411ca0c035891c95121e" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 46577 } }
I20260925 07:32:51.473610 23286 sys_catalog.cc:455] T 00000000000000000000000000000000 P 265a353e5527411ca0c035891c95121e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 265a353e5527411ca0c035891c95121e. Latest consensus state: current_term: 1 leader_uuid: "265a353e5527411ca0c035891c95121e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "265a353e5527411ca0c035891c95121e" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 46577 } } }
I20260925 07:32:51.473691 23286 sys_catalog.cc:458] T 00000000000000000000000000000000 P 265a353e5527411ca0c035891c95121e [sys.catalog]: This master's current role is: LEADER
I20260925 07:32:51.474052 23285 sys_catalog.cc:455] T 00000000000000000000000000000000 P 265a353e5527411ca0c035891c95121e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "265a353e5527411ca0c035891c95121e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "265a353e5527411ca0c035891c95121e" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 46577 } } }
I20260925 07:32:51.474123 23285 sys_catalog.cc:458] T 00000000000000000000000000000000 P 265a353e5527411ca0c035891c95121e [sys.catalog]: This master's current role is: LEADER
W20260925 07:32:51.474745 23299 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 265a353e5527411ca0c035891c95121e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 07:32:51.474800 23299 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 07:32:51.474845 23296 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 07:32:51.475018 23296 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 07:32:51.476078 23282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:51.465085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37756 (local address 127.9.66.62:46577)
0925 07:32:51.465252 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 07:32:51.465266 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 07:32:51.465290 (+ 24us) server_negotiation.cc:407] Connection header received
0925 07:32:51.465821 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:32:51.465828 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:32:51.465877 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:32:51.465987 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:32:51.466781 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:32:51.467799 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:32:51.468791 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:32:51.469185 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:32:51.469263 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:32:51.472071 (+ 2808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:32:51.472075 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:32:51.472075 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:32:51.472086 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:32:51.474295 (+ 2209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:32:51.474759 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:32:51.474762 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:32:51.474762 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:32:51.474867 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:32:51.475135 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:32:51.475136 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:32:51.475136 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:32:51.475410 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:32:51.475516 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 07:32:51.475755 (+ 239us) server_negotiation.cc:299] Negotiation successful
0925 07:32:51.475883 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":92,"threads_started":1}
I20260925 07:32:51.476620 23296 catalog_manager.cc:1347] Generated new cluster ID: 0801089512b5455085114a5cfb31516e
I20260925 07:32:51.476675 23296 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 07:32:51.505676 23296 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 07:32:51.506232 23296 catalog_manager.cc:1504] Loading token signing keys...
I20260925 07:32:51.511214 23296 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 265a353e5527411ca0c035891c95121e: Generated new TSK 0
I20260925 07:32:51.511396 23296 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.66.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.66.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.66.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.1.keytab.
Entry for principal kudu/127.9.66.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.1.keytab.
I20260925 07:32:51.543941 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.66.1:0
--local_ip_for_outbound_sockets=127.9.66.1
--webserver_interface=127.9.66.1
--webserver_port=0
--tserver_master_addrs=127.9.66.62:46577
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.1.keytab
--principal=kudu/127.9.66.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:35859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:32:51.641685 23305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:32:51.641947 23305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:32:51.641984 23305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:32:51.643297 23305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:32:51.643338 23305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:32:51.643370 23305 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.66.1
I20260925 07:32:51.644677 23305 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:35859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.1.keytab
--principal=kudu/127.9.66.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.66.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.66.1
--webserver_port=0
--tserver_master_addrs=127.9.66.62:46577
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23305
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.66.1
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:32:51.644834 23305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:32:51.644984 23305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:32:51.649694 23310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:32:51.649683 23311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:32:51.649703 23305 server_base.cc:1034] running on GCE node
W20260925 07:32:51.649662 23313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:32:51.650040 23305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:32:51.650307 23305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:32:51.651453 23305 hybrid_clock.cc:648] HybridClock initialized: now 1790321571651436 us; error 30 us; skew 500 ppm
Sep 25 07:32:51 dist-test-slave-57fn krb5kdc[23198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321571, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:32:51.653786 23305 init.cc:377] Logged in from keytab as kudu/127.9.66.1@KRBTEST.COM (short username kudu)
I20260925 07:32:51.654119 23305 webserver.cc:460] Webserver started at http://127.9.66.1:46549/ using document root <none> and password file <none>
I20260925 07:32:51.654332 23305 fs_manager.cc:359] Metadata directory not provided
I20260925 07:32:51.654374 23305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:32:51.654506 23305 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:32:51.655426 23305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-0/data/instance:
uuid: "0b7b4fae549a422797b977bcf5e09cdc"
format_stamp: "Formatted at 2026-09-25 07:32:51 on dist-test-slave-57fn"
server_key: "85dcf2e91ba7b87705987c903cabf23b"
server_key_iv: "52b33f563b20366aa695e29ac5be35ff"
server_key_version: "encryptionkey@0"
I20260925 07:32:51.655761 23305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-0/wal/instance:
uuid: "0b7b4fae549a422797b977bcf5e09cdc"
format_stamp: "Formatted at 2026-09-25 07:32:51 on dist-test-slave-57fn"
server_key: "85dcf2e91ba7b87705987c903cabf23b"
server_key_iv: "52b33f563b20366aa695e29ac5be35ff"
server_key_version: "encryptionkey@0"
I20260925 07:32:51.656997 23305 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 07:32:51.657680 23320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:32:51.657842 23305 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 07:32:51.657932 23305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-0/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-0/wal
uuid: "0b7b4fae549a422797b977bcf5e09cdc"
format_stamp: "Formatted at 2026-09-25 07:32:51 on dist-test-slave-57fn"
server_key: "85dcf2e91ba7b87705987c903cabf23b"
server_key_iv: "52b33f563b20366aa695e29ac5be35ff"
server_key_version: "encryptionkey@0"
I20260925 07:32:51.658030 23305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:32:51.670116 23305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:32:51.672211 23305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:32:51.672360 23305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:32:51.672580 23305 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:32:51.672906 23305 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:32:51.672946 23305 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:32:51.672981 23305 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:32:51.673007 23305 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:32:51.678476 23305 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.1:37333
I20260925 07:32:51.678524 23433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.1:37333 every 8 connection(s)
I20260925 07:32:51.678792 23305 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-0/data/info.pb
I20260925 07:32:51.679242 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 23305
I20260925 07:32:51.679375 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-0/wal/instance
I20260925 07:32:51.679565 9480 external_mini_cluster.cc:1428] Setting key aff6d8c3318d925d2fb256ba1681d811
Sep 25 07:32:51 dist-test-slave-57fn krb5kdc[23198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321571, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.1@KRBTEST.COM for kudu/127.9.66.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.66.2@KRBTEST.COM; defaulting to no policy
I20260925 07:32:51.688727 23282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:51.679399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.1:44407 (local address 127.9.66.62:46577)
0925 07:32:51.679441 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 07:32:51.679443 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:32:51.679671 (+ 228us) server_negotiation.cc:407] Connection header received
0925 07:32:51.680207 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:32:51.680208 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:32:51.680219 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:32:51.680261 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:32:51.681163 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:32:51.681899 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:32:51.683157 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:32:51.683498 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:32:51.683571 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:32:51.685931 (+ 2360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:32:51.685932 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:32:51.685932 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:32:51.685939 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:32:51.687265 (+ 1326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:32:51.687765 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:32:51.687766 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:32:51.687766 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:32:51.687815 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:32:51.688068 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:32:51.688068 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:32:51.688068 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:32:51.688165 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:32:51.688259 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 07:32:51.688594 (+ 335us) server_negotiation.cc:299] Negotiation successful
0925 07:32:51.688666 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 07:32:51.688839 23436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:51.679392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:46577 (local address 127.9.66.1:44407)
0925 07:32:51.679547 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 07:32:51.679583 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 07:32:51.680084 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:32:51.680321 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:32:51.680325 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:32:51.680886 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:32:51.681064 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:32:51.681069 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:32:51.681996 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:32:51.681997 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:32:51.683061 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:32:51.683064 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:32:51.683592 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:32:51.683593 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:32:51.683686 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:32:51.684171 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:32:51.684178 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:32:51.685804 (+ 1626us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:32:51.687352 (+ 1548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:32:51.687359 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:32:51.687361 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:32:51.687682 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:32:51.687882 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:32:51.687882 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:32:51.687883 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:32:51.688007 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:32:51.688269 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:32:51.688276 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:32:51.688438 (+ 162us) client_negotiation.cc:769] Sending connection context
0925 07:32:51.688555 (+ 117us) client_negotiation.cc:240] Negotiation successful
0925 07:32:51.688668 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":98,"threads_started":1}
I20260925 07:32:51.689388 23434 heartbeater.cc:347] Connected to a master server at 127.9.66.62:46577
I20260925 07:32:51.689463 23434 heartbeater.cc:464] Registering TS with master...
I20260925 07:32:51.689656 23434 heartbeater.cc:511] Master 127.9.66.62:46577 requested a full tablet report, sending...
I20260925 07:32:51.690166 23243 ts_manager.cc:194] Registered new tserver with Master: 0b7b4fae549a422797b977bcf5e09cdc (127.9.66.1:37333)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.66.2@KRBTEST.COM" created.
I20260925 07:32:51.690920 23243 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.66.1@KRBTEST.COM'} at 127.9.66.1:44407
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.66.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.2.keytab.
Entry for principal kudu/127.9.66.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.2.keytab.
I20260925 07:32:51.704582 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.66.2:0
--local_ip_for_outbound_sockets=127.9.66.2
--webserver_interface=127.9.66.2
--webserver_port=0
--tserver_master_addrs=127.9.66.62:46577
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.2.keytab
--principal=kudu/127.9.66.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:35859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:32:51.783258 23439 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:32:51.783455 23439 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:32:51.783478 23439 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:32:51.784665 23439 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:32:51.784708 23439 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:32:51.784734 23439 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.66.2
I20260925 07:32:51.786187 23439 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:35859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.2.keytab
--principal=kudu/127.9.66.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.66.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.66.2
--webserver_port=0
--tserver_master_addrs=127.9.66.62:46577
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23439
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.66.2
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:32:51.786412 23439 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:32:51.786643 23439 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:32:51.791550 23444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:32:51.791608 23447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:32:51.791620 23439 server_base.cc:1034] running on GCE node
W20260925 07:32:51.791646 23445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:32:51.792019 23439 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:32:51.792261 23439 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:32:51.793423 23439 hybrid_clock.cc:648] HybridClock initialized: now 1790321571793419 us; error 37 us; skew 500 ppm
Sep 25 07:32:51 dist-test-slave-57fn krb5kdc[23198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321571, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:32:51.795892 23439 init.cc:377] Logged in from keytab as kudu/127.9.66.2@KRBTEST.COM (short username kudu)
I20260925 07:32:51.796319 23439 webserver.cc:460] Webserver started at http://127.9.66.2:35775/ using document root <none> and password file <none>
I20260925 07:32:51.796490 23439 fs_manager.cc:359] Metadata directory not provided
I20260925 07:32:51.796530 23439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:32:51.796615 23439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:32:51.797605 23439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-1/data/instance:
uuid: "c59f513dd13b45f4a554c64ff7a03c87"
format_stamp: "Formatted at 2026-09-25 07:32:51 on dist-test-slave-57fn"
server_key: "f2804f27d213bc1967c47cb055e357d5"
server_key_iv: "0cc2b036bc76740d6e03c407966f7a62"
server_key_version: "encryptionkey@0"
I20260925 07:32:51.797901 23439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-1/wal/instance:
uuid: "c59f513dd13b45f4a554c64ff7a03c87"
format_stamp: "Formatted at 2026-09-25 07:32:51 on dist-test-slave-57fn"
server_key: "f2804f27d213bc1967c47cb055e357d5"
server_key_iv: "0cc2b036bc76740d6e03c407966f7a62"
server_key_version: "encryptionkey@0"
I20260925 07:32:51.799095 23439 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:32:51.799777 23454 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:32:51.800020 23439 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:32:51.800104 23439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-1/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-1/wal
uuid: "c59f513dd13b45f4a554c64ff7a03c87"
format_stamp: "Formatted at 2026-09-25 07:32:51 on dist-test-slave-57fn"
server_key: "f2804f27d213bc1967c47cb055e357d5"
server_key_iv: "0cc2b036bc76740d6e03c407966f7a62"
server_key_version: "encryptionkey@0"
I20260925 07:32:51.800149 23439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:32:51.817687 23439 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:32:51.819860 23439 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:32:51.819965 23439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:32:51.820158 23439 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:32:51.820447 23439 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:32:51.820485 23439 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:32:51.820506 23439 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:32:51.820519 23439 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:32:51.826556 23439 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.2:38723
I20260925 07:32:51.826622 23567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.2:38723 every 8 connection(s)
I20260925 07:32:51.826936 23439 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-1/data/info.pb
I20260925 07:32:51.829058 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 23439
I20260925 07:32:51.829169 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-1/wal/instance
I20260925 07:32:51.829350 9480 external_mini_cluster.cc:1428] Setting key d8aa650df83996334dee569a7fc97dff
Sep 25 07:32:51 dist-test-slave-57fn krb5kdc[23198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321571, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.2@KRBTEST.COM for kudu/127.9.66.62@KRBTEST.COM
I20260925 07:32:51.837277 23282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:51.827628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.2:57803 (local address 127.9.66.62:46577)
0925 07:32:51.827680 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 07:32:51.827684 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 07:32:51.827864 (+ 180us) server_negotiation.cc:407] Connection header received
0925 07:32:51.828366 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:32:51.828367 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:32:51.828375 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:32:51.828430 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:32:51.829555 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:32:51.830165 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:32:51.831677 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:32:51.831992 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:32:51.832068 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:32:51.834369 (+ 2301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:32:51.834371 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:32:51.834371 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:32:51.834379 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:32:51.835995 (+ 1616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:32:51.836352 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:32:51.836353 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:32:51.836354 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:32:51.836418 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:32:51.836633 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:32:51.836634 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:32:51.836635 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:32:51.836776 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:32:51.836856 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 07:32:51.837092 (+ 236us) server_negotiation.cc:299] Negotiation successful
0925 07:32:51.837170 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 07:32:51.837404 23570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:51.827595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:46577 (local address 127.9.66.2:57803)
0925 07:32:51.827751 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 07:32:51.827791 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 07:32:51.828282 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:32:51.828498 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:32:51.828501 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:32:51.829161 (+ 660us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:32:51.829437 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:32:51.829444 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:32:51.830266 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:32:51.830267 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:32:51.831581 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:32:51.831585 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:32:51.832076 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:32:51.832076 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:32:51.832174 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:32:51.832697 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:32:51.832704 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:32:51.834276 (+ 1572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:32:51.836082 (+ 1806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:32:51.836088 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:32:51.836089 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:32:51.836285 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:32:51.836488 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:32:51.836488 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:32:51.836489 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:32:51.836572 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:32:51.836843 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:32:51.836845 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:32:51.836939 (+ 94us) client_negotiation.cc:769] Sending connection context
0925 07:32:51.837062 (+ 123us) client_negotiation.cc:240] Negotiation successful
0925 07:32:51.837220 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":92,"threads_started":1}
I20260925 07:32:51.837855 23568 heartbeater.cc:347] Connected to a master server at 127.9.66.62:46577
I20260925 07:32:51.837941 23568 heartbeater.cc:464] Registering TS with master...
I20260925 07:32:51.838182 23568 heartbeater.cc:511] Master 127.9.66.62:46577 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.9.66.3@KRBTEST.COM; defaulting to no policy
I20260925 07:32:51.838588 23243 ts_manager.cc:194] Registered new tserver with Master: c59f513dd13b45f4a554c64ff7a03c87 (127.9.66.2:38723)
I20260925 07:32:51.839013 23243 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.66.2@KRBTEST.COM'} at 127.9.66.2:57803
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.66.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.66.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.3.keytab.
Entry for principal kudu/127.9.66.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.3.keytab.
I20260925 07:32:51.855620 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.66.3:0
--local_ip_for_outbound_sockets=127.9.66.3
--webserver_interface=127.9.66.3
--webserver_port=0
--tserver_master_addrs=127.9.66.62:46577
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.3.keytab
--principal=kudu/127.9.66.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:35859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:32:51.933907 23573 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:32:51.934113 23573 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:32:51.934135 23573 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:32:51.935307 23573 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:32:51.935346 23573 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:32:51.935371 23573 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.66.3
I20260925 07:32:51.936717 23573 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:35859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.3.keytab
--principal=kudu/127.9.66.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.66.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.66.3
--webserver_port=0
--tserver_master_addrs=127.9.66.62:46577
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23573
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.66.3
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:32:51.936914 23573 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:32:51.937137 23573 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:32:51.941910 23578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:32:51.941931 23581 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:32:51.941928 23579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:32:51.942073 23573 server_base.cc:1034] running on GCE node
I20260925 07:32:51.942269 23573 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:32:51.942492 23573 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:32:51.943645 23573 hybrid_clock.cc:648] HybridClock initialized: now 1790321571943629 us; error 29 us; skew 500 ppm
Sep 25 07:32:51 dist-test-slave-57fn krb5kdc[23198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321571, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:32:51.945994 23573 init.cc:377] Logged in from keytab as kudu/127.9.66.3@KRBTEST.COM (short username kudu)
I20260925 07:32:51.946412 23573 webserver.cc:460] Webserver started at http://127.9.66.3:45283/ using document root <none> and password file <none>
I20260925 07:32:51.946630 23573 fs_manager.cc:359] Metadata directory not provided
I20260925 07:32:51.946695 23573 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:32:51.946833 23573 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:32:51.947839 23573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-2/data/instance:
uuid: "74c3871e1dde4f84bda59faeceb249fe"
format_stamp: "Formatted at 2026-09-25 07:32:51 on dist-test-slave-57fn"
server_key: "f3df166509c3aac269bd51f1d78d232d"
server_key_iv: "4cccb95ed696f0c469ea187ac84a5f5a"
server_key_version: "encryptionkey@0"
I20260925 07:32:51.948203 23573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-2/wal/instance:
uuid: "74c3871e1dde4f84bda59faeceb249fe"
format_stamp: "Formatted at 2026-09-25 07:32:51 on dist-test-slave-57fn"
server_key: "f3df166509c3aac269bd51f1d78d232d"
server_key_iv: "4cccb95ed696f0c469ea187ac84a5f5a"
server_key_version: "encryptionkey@0"
I20260925 07:32:51.949594 23573 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 07:32:51.950415 23588 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:32:51.950690 23573 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:32:51.950789 23573 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-2/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-2/wal
uuid: "74c3871e1dde4f84bda59faeceb249fe"
format_stamp: "Formatted at 2026-09-25 07:32:51 on dist-test-slave-57fn"
server_key: "f3df166509c3aac269bd51f1d78d232d"
server_key_iv: "4cccb95ed696f0c469ea187ac84a5f5a"
server_key_version: "encryptionkey@0"
I20260925 07:32:51.950873 23573 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:32:51.966533 23573 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:32:51.968298 23573 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:32:51.968451 23573 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:32:51.968669 23573 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:32:51.968948 23573 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:32:51.969002 23573 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:32:51.969041 23573 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:32:51.969069 23573 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:32:51.975304 23573 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.3:43023
I20260925 07:32:51.975338 23701 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.3:43023 every 8 connection(s)
I20260925 07:32:51.975633 23573 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-2/data/info.pb
I20260925 07:32:51.979686 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 23573
I20260925 07:32:51.979761 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-2/wal/instance
I20260925 07:32:51.979916 9480 external_mini_cluster.cc:1428] Setting key d9f53c4f23e980e843977bdbfda70907
Sep 25 07:32:51 dist-test-slave-57fn krb5kdc[23198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321571, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.3@KRBTEST.COM for kudu/127.9.66.62@KRBTEST.COM
I20260925 07:32:51.984578 23282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:51.976315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.3:47371 (local address 127.9.66.62:46577)
0925 07:32:51.976400 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0925 07:32:51.976402 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:32:51.976565 (+ 163us) server_negotiation.cc:407] Connection header received
0925 07:32:51.977077 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:32:51.977078 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:32:51.977088 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:32:51.977152 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:32:51.978100 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:32:51.978590 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:32:51.979549 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:32:51.979726 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:32:51.979820 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:32:51.982128 (+ 2308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:32:51.982129 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:32:51.982129 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:32:51.982134 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:32:51.983383 (+ 1249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:32:51.983751 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:32:51.983752 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:32:51.983753 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:32:51.983821 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:32:51.984053 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:32:51.984054 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:32:51.984054 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:32:51.984156 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:32:51.984208 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 07:32:51.984453 (+ 245us) server_negotiation.cc:299] Negotiation successful
0925 07:32:51.984517 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260925 07:32:51.984697 23704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:51.976289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:46577 (local address 127.9.66.3:47371)
0925 07:32:51.976439 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 07:32:51.976478 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 07:32:51.976955 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:32:51.977218 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:32:51.977221 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:32:51.977813 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:32:51.977996 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:32:51.978000 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:32:51.978672 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:32:51.978673 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:32:51.979454 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:32:51.979456 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:32:51.979844 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:32:51.979845 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:32:51.979993 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:32:51.980574 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:32:51.980584 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:32:51.982049 (+ 1465us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:32:51.983452 (+ 1403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:32:51.983458 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:32:51.983459 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:32:51.983664 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:32:51.983896 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:32:51.983896 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:32:51.983897 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:32:51.983995 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:32:51.984212 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:32:51.984215 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:32:51.984326 (+ 111us) client_negotiation.cc:769] Sending connection context
0925 07:32:51.984403 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 07:32:51.984521 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":90,"threads_started":1}
I20260925 07:32:51.985186 23702 heartbeater.cc:347] Connected to a master server at 127.9.66.62:46577
I20260925 07:32:51.985262 23702 heartbeater.cc:464] Registering TS with master...
I20260925 07:32:51.985445 23702 heartbeater.cc:511] Master 127.9.66.62:46577 requested a full tablet report, sending...
I20260925 07:32:51.985836 23243 ts_manager.cc:194] Registered new tserver with Master: 74c3871e1dde4f84bda59faeceb249fe (127.9.66.3:43023)
I20260925 07:32:51.986207 23243 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.66.3@KRBTEST.COM'} at 127.9.66.3:47371
I20260925 07:32:51.991539 9480 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 07:32:51.998485 23282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:51.992736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37762 (local address 127.9.66.62:46577)
0925 07:32:51.992800 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 07:32:51.992802 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:32:51.992908 (+ 106us) server_negotiation.cc:407] Connection header received
0925 07:32:51.992967 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:32:51.992967 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:32:51.992977 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:32:51.993031 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:32:51.993588 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:32:51.994030 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:32:51.994746 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:32:51.994927 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:32:51.994997 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:32:51.996091 (+ 1094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:32:51.996093 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:32:51.996093 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:32:51.996099 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:32:51.997290 (+ 1191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:32:51.997690 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:32:51.997691 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:32:51.997691 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:32:51.997765 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:32:51.997963 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:32:51.997964 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:32:51.997964 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:32:51.998104 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:32:51.998210 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 07:32:51.998306 (+ 96us) server_negotiation.cc:299] Negotiation successful
0925 07:32:51.998382 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 07:32:52.000054 23243 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37762:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 07:32:52.001636 23243 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 07:32:52.007853 23713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:52.003971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37232 (local address 127.9.66.2:38723)
0925 07:32:52.004132 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 07:32:52.004136 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 07:32:52.004146 (+ 10us) server_negotiation.cc:407] Connection header received
0925 07:32:52.004185 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:32:52.004187 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:32:52.004204 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:32:52.004288 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:32:52.004612 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:32:52.005736 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:32:52.006964 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:32:52.007469 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:32:52.007560 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:32:52.007620 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 07:32:52.007735 (+ 115us) server_negotiation.cc:299] Negotiation successful
0925 07:32:52.007777 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":94,"threads_started":1}
I20260925 07:32:52.007846 23711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:52.003727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.2:38723 (local address 127.0.0.1:37232)
0925 07:32:52.003847 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0925 07:32:52.003885 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 07:32:52.003946 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:32:52.004308 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:32:52.004311 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:32:52.004314 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:32:52.004377 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:32:52.004378 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:32:52.005835 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:32:52.005837 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:32:52.006880 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:32:52.006881 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:32:52.007559 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:32:52.007560 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:32:52.007637 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:32:52.007649 (+ 12us) client_negotiation.cc:769] Sending connection context
0925 07:32:52.007721 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 07:32:52.007773 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"mutex_wait_us":1,"thread_start_us":78,"threads_started":1}
I20260925 07:32:52.008013 23714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:52.003976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50828 (local address 127.9.66.1:37333)
0925 07:32:52.004291 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0925 07:32:52.004295 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 07:32:52.004303 (+ 8us) server_negotiation.cc:407] Connection header received
0925 07:32:52.004337 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:32:52.004339 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:32:52.004353 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:32:52.004433 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:32:52.004572 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:32:52.005736 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:32:52.006829 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:32:52.007544 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:32:52.007599 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:32:52.007643 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0925 07:32:52.007903 (+ 260us) server_negotiation.cc:299] Negotiation successful
0925 07:32:52.007939 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":96,"threads_started":1}
I20260925 07:32:52.008008 23715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:52.004157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53652 (local address 127.9.66.3:43023)
0925 07:32:52.004316 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0925 07:32:52.004320 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 07:32:52.004328 (+ 8us) server_negotiation.cc:407] Connection header received
0925 07:32:52.004359 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:32:52.004361 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:32:52.004374 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:32:52.004437 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:32:52.004627 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:32:52.005331 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:32:52.006778 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:32:52.007501 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:32:52.007584 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:32:52.007640 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 07:32:52.007890 (+ 250us) server_negotiation.cc:299] Negotiation successful
0925 07:32:52.007931 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":72,"threads_started":1}
I20260925 07:32:52.008169 23710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:52.003728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.1:37333 (local address 127.0.0.1:50828)
0925 07:32:52.004121 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0925 07:32:52.004130 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 07:32:52.004161 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:32:52.004430 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:32:52.004431 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:32:52.004433 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:32:52.004485 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:32:52.004487 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:32:52.005862 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:32:52.005864 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:32:52.006716 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:32:52.006718 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:32:52.007679 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:32:52.007680 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:32:52.007749 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:32:52.007753 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 07:32:52.007803 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 07:32:52.007847 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384,"mutex_wait_us":6,"thread_start_us":63,"threads_started":1}
I20260925 07:32:52.008311 23712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:52.003728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.3:43023 (local address 127.0.0.1:53652)
0925 07:32:52.003992 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0925 07:32:52.004001 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 07:32:52.004053 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:32:52.004506 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:32:52.004507 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:32:52.004508 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:32:52.004560 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:32:52.004561 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:32:52.005433 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:32:52.005434 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:32:52.006698 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:32:52.006701 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:32:52.007678 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:32:52.007679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:32:52.007755 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:32:52.007758 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 07:32:52.007803 (+ 45us) client_negotiation.cc:240] Negotiation successful
0925 07:32:52.007846 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"mutex_wait_us":3,"thread_start_us":61,"threads_started":1}
I20260925 07:32:52.008953 23636 tablet_service.cc:1467] Processing CreateTablet for tablet 01813c803dd8444596df2261f1e94ef2 (DEFAULT_TABLE table=test-table [id=36c3b7969d1a42cc8f2992aad6751705]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:32:52.008955 23368 tablet_service.cc:1467] Processing CreateTablet for tablet 01813c803dd8444596df2261f1e94ef2 (DEFAULT_TABLE table=test-table [id=36c3b7969d1a42cc8f2992aad6751705]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:32:52.008953 23502 tablet_service.cc:1467] Processing CreateTablet for tablet 01813c803dd8444596df2261f1e94ef2 (DEFAULT_TABLE table=test-table [id=36c3b7969d1a42cc8f2992aad6751705]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:32:52.009330 23636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01813c803dd8444596df2261f1e94ef2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:32:52.009326 23502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01813c803dd8444596df2261f1e94ef2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:32:52.009326 23368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01813c803dd8444596df2261f1e94ef2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:32:52.011940 23716 tablet_bootstrap.cc:492] T 01813c803dd8444596df2261f1e94ef2 P 74c3871e1dde4f84bda59faeceb249fe: Bootstrap starting.
I20260925 07:32:52.012615 23716 tablet_bootstrap.cc:654] T 01813c803dd8444596df2261f1e94ef2 P 74c3871e1dde4f84bda59faeceb249fe: Neither blocks nor log segments found. Creating new log.
I20260925 07:32:52.012735 23717 tablet_bootstrap.cc:492] T 01813c803dd8444596df2261f1e94ef2 P c59f513dd13b45f4a554c64ff7a03c87: Bootstrap starting.
I20260925 07:32:52.013166 23716 log.cc:824] T 01813c803dd8444596df2261f1e94ef2 P 74c3871e1dde4f84bda59faeceb249fe: Log is configured to *not* fsync() on all Append() calls
I20260925 07:32:52.013341 23717 tablet_bootstrap.cc:654] T 01813c803dd8444596df2261f1e94ef2 P c59f513dd13b45f4a554c64ff7a03c87: Neither blocks nor log segments found. Creating new log.
I20260925 07:32:52.013701 23718 tablet_bootstrap.cc:492] T 01813c803dd8444596df2261f1e94ef2 P 0b7b4fae549a422797b977bcf5e09cdc: Bootstrap starting.
I20260925 07:32:52.013964 23717 log.cc:824] T 01813c803dd8444596df2261f1e94ef2 P c59f513dd13b45f4a554c64ff7a03c87: Log is configured to *not* fsync() on all Append() calls
I20260925 07:32:52.014073 23716 tablet_bootstrap.cc:492] T 01813c803dd8444596df2261f1e94ef2 P 74c3871e1dde4f84bda59faeceb249fe: No bootstrap required, opened a new log
I20260925 07:32:52.014158 23716 ts_tablet_manager.cc:1397] T 01813c803dd8444596df2261f1e94ef2 P 74c3871e1dde4f84bda59faeceb249fe: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 07:32:52.014424 23718 tablet_bootstrap.cc:654] T 01813c803dd8444596df2261f1e94ef2 P 0b7b4fae549a422797b977bcf5e09cdc: Neither blocks nor log segments found. Creating new log.
I20260925 07:32:52.014665 23717 tablet_bootstrap.cc:492] T 01813c803dd8444596df2261f1e94ef2 P c59f513dd13b45f4a554c64ff7a03c87: No bootstrap required, opened a new log
I20260925 07:32:52.014742 23717 ts_tablet_manager.cc:1397] T 01813c803dd8444596df2261f1e94ef2 P c59f513dd13b45f4a554c64ff7a03c87: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 07:32:52.015018 23718 log.cc:824] T 01813c803dd8444596df2261f1e94ef2 P 0b7b4fae549a422797b977bcf5e09cdc: Log is configured to *not* fsync() on all Append() calls
I20260925 07:32:52.015820 23718 tablet_bootstrap.cc:492] T 01813c803dd8444596df2261f1e94ef2 P 0b7b4fae549a422797b977bcf5e09cdc: No bootstrap required, opened a new log
I20260925 07:32:52.015910 23718 ts_tablet_manager.cc:1397] T 01813c803dd8444596df2261f1e94ef2 P 0b7b4fae549a422797b977bcf5e09cdc: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 07:32:52.016425 23717 raft_consensus.cc:348] T 01813c803dd8444596df2261f1e94ef2 P c59f513dd13b45f4a554c64ff7a03c87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c3871e1dde4f84bda59faeceb249fe" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 43023 } } peers { permanent_uuid: "0b7b4fae549a422797b977bcf5e09cdc" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 37333 } } peers { permanent_uuid: "c59f513dd13b45f4a554c64ff7a03c87" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 38723 } }
I20260925 07:32:52.016420 23716 raft_consensus.cc:348] T 01813c803dd8444596df2261f1e94ef2 P 74c3871e1dde4f84bda59faeceb249fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c3871e1dde4f84bda59faeceb249fe" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 43023 } } peers { permanent_uuid: "0b7b4fae549a422797b977bcf5e09cdc" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 37333 } } peers { permanent_uuid: "c59f513dd13b45f4a554c64ff7a03c87" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 38723 } }
I20260925 07:32:52.016585 23716 raft_consensus.cc:374] T 01813c803dd8444596df2261f1e94ef2 P 74c3871e1dde4f84bda59faeceb249fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:32:52.016585 23717 raft_consensus.cc:374] T 01813c803dd8444596df2261f1e94ef2 P c59f513dd13b45f4a554c64ff7a03c87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:32:52.016631 23717 raft_consensus.cc:729] T 01813c803dd8444596df2261f1e94ef2 P c59f513dd13b45f4a554c64ff7a03c87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c59f513dd13b45f4a554c64ff7a03c87, State: Initialized, Role: FOLLOWER
I20260925 07:32:52.016661 23716 raft_consensus.cc:729] T 01813c803dd8444596df2261f1e94ef2 P 74c3871e1dde4f84bda59faeceb249fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74c3871e1dde4f84bda59faeceb249fe, State: Initialized, Role: FOLLOWER
I20260925 07:32:52.016777 23717 consensus_queue.cc:260] T 01813c803dd8444596df2261f1e94ef2 P c59f513dd13b45f4a554c64ff7a03c87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c3871e1dde4f84bda59faeceb249fe" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 43023 } } peers { permanent_uuid: "0b7b4fae549a422797b977bcf5e09cdc" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 37333 } } peers { permanent_uuid: "c59f513dd13b45f4a554c64ff7a03c87" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 38723 } }
I20260925 07:32:52.016790 23716 consensus_queue.cc:260] T 01813c803dd8444596df2261f1e94ef2 P 74c3871e1dde4f84bda59faeceb249fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c3871e1dde4f84bda59faeceb249fe" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 43023 } } peers { permanent_uuid: "0b7b4fae549a422797b977bcf5e09cdc" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 37333 } } peers { permanent_uuid: "c59f513dd13b45f4a554c64ff7a03c87" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 38723 } }
I20260925 07:32:52.017083 23716 ts_tablet_manager.cc:1428] T 01813c803dd8444596df2261f1e94ef2 P 74c3871e1dde4f84bda59faeceb249fe: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 07:32:52.017091 23717 ts_tablet_manager.cc:1428] T 01813c803dd8444596df2261f1e94ef2 P c59f513dd13b45f4a554c64ff7a03c87: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260925 07:32:52.017103 23702 heartbeater.cc:503] Master 127.9.66.62:46577 was elected leader, sending a full tablet report...
I20260925 07:32:52.017208 23568 heartbeater.cc:503] Master 127.9.66.62:46577 was elected leader, sending a full tablet report...
I20260925 07:32:52.018069 23718 raft_consensus.cc:348] T 01813c803dd8444596df2261f1e94ef2 P 0b7b4fae549a422797b977bcf5e09cdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c3871e1dde4f84bda59faeceb249fe" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 43023 } } peers { permanent_uuid: "0b7b4fae549a422797b977bcf5e09cdc" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 37333 } } peers { permanent_uuid: "c59f513dd13b45f4a554c64ff7a03c87" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 38723 } }
I20260925 07:32:52.018244 23718 raft_consensus.cc:374] T 01813c803dd8444596df2261f1e94ef2 P 0b7b4fae549a422797b977bcf5e09cdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:32:52.018334 23718 raft_consensus.cc:729] T 01813c803dd8444596df2261f1e94ef2 P 0b7b4fae549a422797b977bcf5e09cdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b7b4fae549a422797b977bcf5e09cdc, State: Initialized, Role: FOLLOWER
I20260925 07:32:52.018460 23718 consensus_queue.cc:260] T 01813c803dd8444596df2261f1e94ef2 P 0b7b4fae549a422797b977bcf5e09cdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c3871e1dde4f84bda59faeceb249fe" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 43023 } } peers { permanent_uuid: "0b7b4fae549a422797b977bcf5e09cdc" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 37333 } } peers { permanent_uuid: "c59f513dd13b45f4a554c64ff7a03c87" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 38723 } }
I20260925 07:32:52.018733 23718 ts_tablet_manager.cc:1428] T 01813c803dd8444596df2261f1e94ef2 P 0b7b4fae549a422797b977bcf5e09cdc: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 07:32:52.018764 23434 heartbeater.cc:503] Master 127.9.66.62:46577 was elected leader, sending a full tablet report...
W20260925 07:32:52.077567 23569 tablet.cc:2369] T 01813c803dd8444596df2261f1e94ef2 P c59f513dd13b45f4a554c64ff7a03c87: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 07:32:52.179498 23435 tablet.cc:2369] T 01813c803dd8444596df2261f1e94ef2 P 0b7b4fae549a422797b977bcf5e09cdc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 07:32:52.226174 23703 tablet.cc:2369] T 01813c803dd8444596df2261f1e94ef2 P 74c3871e1dde4f84bda59faeceb249fe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 07:32:52.278162 23722 raft_consensus.cc:482] T 01813c803dd8444596df2261f1e94ef2 P c59f513dd13b45f4a554c64ff7a03c87 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 07:32:52.278293 23722 raft_consensus.cc:504] T 01813c803dd8444596df2261f1e94ef2 P c59f513dd13b45f4a554c64ff7a03c87 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c3871e1dde4f84bda59faeceb249fe" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 43023 } } peers { permanent_uuid: "0b7b4fae549a422797b977bcf5e09cdc" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 37333 } } peers { permanent_uuid: "c59f513dd13b45f4a554c64ff7a03c87" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 38723 } }
I20260925 07:32:52.278731 23722 leader_election.cc:288] T 01813c803dd8444596df2261f1e94ef2 P c59f513dd13b45f4a554c64ff7a03c87 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74c3871e1dde4f84bda59faeceb249fe (127.9.66.3:43023), 0b7b4fae549a422797b977bcf5e09cdc (127.9.66.1:37333)
I20260925 07:32:52.282693 23715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:52.279074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.2:47875 (local address 127.9.66.3:43023)
0925 07:32:52.279122 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 07:32:52.279124 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:32:52.279134 (+ 10us) server_negotiation.cc:407] Connection header received
0925 07:32:52.279166 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:32:52.279166 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:32:52.279183 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:32:52.279277 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:32:52.279509 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:32:52.280241 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:32:52.281753 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:32:52.282314 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:32:52.282390 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:32:52.282401 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 07:32:52.282588 (+ 187us) server_negotiation.cc:299] Negotiation successful
0925 07:32:52.282625 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 07:32:52.282693 23714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:52.279074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.2:37043 (local address 127.9.66.1:37333)
0925 07:32:52.279122 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 07:32:52.279124 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:32:52.279235 (+ 111us) server_negotiation.cc:407] Connection header received
0925 07:32:52.279263 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:32:52.279264 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:32:52.279276 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:32:52.279331 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:32:52.279499 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:32:52.280247 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:32:52.281753 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:32:52.282314 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:32:52.282390 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:32:52.282401 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 07:32:52.282588 (+ 187us) server_negotiation.cc:299] Negotiation successful
0925 07:32:52.282625 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 07:32:52.282704 23570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:52.278968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.3:43023 (local address 127.9.66.2:47875)
0925 07:32:52.279031 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0925 07:32:52.279044 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 07:32:52.279089 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:32:52.279303 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:32:52.279304 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:32:52.279305 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:32:52.279402 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:32:52.279405 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:32:52.280345 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:32:52.280346 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:32:52.281656 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:32:52.281660 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:32:52.282411 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:32:52.282412 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:32:52.282496 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:32:52.282504 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 07:32:52.282565 (+ 61us) client_negotiation.cc:240] Negotiation successful
0925 07:32:52.282610 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":88}
I20260925 07:32:52.282711 23725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:52.278968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.1:37333 (local address 127.9.66.2:37043)
0925 07:32:52.279160 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0925 07:32:52.279171 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 07:32:52.279219 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:32:52.279346 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:32:52.279346 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:32:52.279348 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:32:52.279421 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:32:52.279423 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:32:52.280345 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:32:52.280346 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:32:52.281657 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:32:52.281660 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:32:52.282426 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:32:52.282427 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:32:52.282496 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:32:52.282504 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 07:32:52.282565 (+ 61us) client_negotiation.cc:240] Negotiation successful
0925 07:32:52.282610 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"mutex_wait_us":21,"thread_start_us":118,"threads_started":1}
I20260925 07:32:52.283105 23656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01813c803dd8444596df2261f1e94ef2" candidate_uuid: "c59f513dd13b45f4a554c64ff7a03c87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74c3871e1dde4f84bda59faeceb249fe" is_pre_election: true
I20260925 07:32:52.283232 23656 raft_consensus.cc:2454] T 01813c803dd8444596df2261f1e94ef2 P 74c3871e1dde4f84bda59faeceb249fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c59f513dd13b45f4a554c64ff7a03c87 in term 0.
I20260925 07:32:52.283177 23388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01813c803dd8444596df2261f1e94ef2" candidate_uuid: "c59f513dd13b45f4a554c64ff7a03c87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b7b4fae549a422797b977bcf5e09cdc" is_pre_election: true
I20260925 07:32:52.283373 23388 raft_consensus.cc:2454] T 01813c803dd8444596df2261f1e94ef2 P 0b7b4fae549a422797b977bcf5e09cdc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c59f513dd13b45f4a554c64ff7a03c87 in term 0.
I20260925 07:32:52.283479 23456 leader_election.cc:302] T 01813c803dd8444596df2261f1e94ef2 P c59f513dd13b45f4a554c64ff7a03c87 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74c3871e1dde4f84bda59faeceb249fe, c59f513dd13b45f4a554c64ff7a03c87; no voters:
I20260925 07:32:52.283604 23722 raft_consensus.cc:2781] T 01813c803dd8444596df2261f1e94ef2 P c59f513dd13b45f4a554c64ff7a03c87 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 07:32:52.283658 23722 raft_consensus.cc:482] T 01813c803dd8444596df2261f1e94ef2 P c59f513dd13b45f4a554c64ff7a03c87 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 07:32:52.283696 23722 raft_consensus.cc:3037] T 01813c803dd8444596df2261f1e94ef2 P c59f513dd13b45f4a554c64ff7a03c87 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:32:52.284523 23722 raft_consensus.cc:504] T 01813c803dd8444596df2261f1e94ef2 P c59f513dd13b45f4a554c64ff7a03c87 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c3871e1dde4f84bda59faeceb249fe" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 43023 } } peers { permanent_uuid: "0b7b4fae549a422797b977bcf5e09cdc" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 37333 } } peers { permanent_uuid: "c59f513dd13b45f4a554c64ff7a03c87" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 38723 } }
I20260925 07:32:52.284662 23722 leader_election.cc:288] T 01813c803dd8444596df2261f1e94ef2 P c59f513dd13b45f4a554c64ff7a03c87 [CANDIDATE]: Term 1 election: Requested vote from peers 74c3871e1dde4f84bda59faeceb249fe (127.9.66.3:43023), 0b7b4fae549a422797b977bcf5e09cdc (127.9.66.1:37333)
I20260925 07:32:52.284817 23656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01813c803dd8444596df2261f1e94ef2" candidate_uuid: "c59f513dd13b45f4a554c64ff7a03c87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74c3871e1dde4f84bda59faeceb249fe"
I20260925 07:32:52.284910 23656 raft_consensus.cc:3037] T 01813c803dd8444596df2261f1e94ef2 P 74c3871e1dde4f84bda59faeceb249fe [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:32:52.285586 23656 raft_consensus.cc:2454] T 01813c803dd8444596df2261f1e94ef2 P 74c3871e1dde4f84bda59faeceb249fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c59f513dd13b45f4a554c64ff7a03c87 in term 1.
I20260925 07:32:52.285765 23456 leader_election.cc:302] T 01813c803dd8444596df2261f1e94ef2 P c59f513dd13b45f4a554c64ff7a03c87 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74c3871e1dde4f84bda59faeceb249fe, c59f513dd13b45f4a554c64ff7a03c87; no voters:
I20260925 07:32:52.285720 23388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01813c803dd8444596df2261f1e94ef2" candidate_uuid: "c59f513dd13b45f4a554c64ff7a03c87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b7b4fae549a422797b977bcf5e09cdc"
I20260925 07:32:52.285833 23388 raft_consensus.cc:3037] T 01813c803dd8444596df2261f1e94ef2 P 0b7b4fae549a422797b977bcf5e09cdc [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:32:52.285920 23722 raft_consensus.cc:2781] T 01813c803dd8444596df2261f1e94ef2 P c59f513dd13b45f4a554c64ff7a03c87 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:32:52.286087 23722 raft_consensus.cc:686] T 01813c803dd8444596df2261f1e94ef2 P c59f513dd13b45f4a554c64ff7a03c87 [term 1 LEADER]: Becoming Leader. State: Replica: c59f513dd13b45f4a554c64ff7a03c87, State: Running, Role: LEADER
I20260925 07:32:52.286180 23722 consensus_queue.cc:237] T 01813c803dd8444596df2261f1e94ef2 P c59f513dd13b45f4a554c64ff7a03c87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c3871e1dde4f84bda59faeceb249fe" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 43023 } } peers { permanent_uuid: "0b7b4fae549a422797b977bcf5e09cdc" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 37333 } } peers { permanent_uuid: "c59f513dd13b45f4a554c64ff7a03c87" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 38723 } }
I20260925 07:32:52.286759 23388 raft_consensus.cc:2454] T 01813c803dd8444596df2261f1e94ef2 P 0b7b4fae549a422797b977bcf5e09cdc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c59f513dd13b45f4a554c64ff7a03c87 in term 1.
I20260925 07:32:52.286985 23243 catalog_manager.cc:5495] T 01813c803dd8444596df2261f1e94ef2 P c59f513dd13b45f4a554c64ff7a03c87 reported cstate change: term changed from 0 to 1, leader changed from <none> to c59f513dd13b45f4a554c64ff7a03c87 (127.9.66.2). New cstate: current_term: 1 leader_uuid: "c59f513dd13b45f4a554c64ff7a03c87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c3871e1dde4f84bda59faeceb249fe" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 43023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b7b4fae549a422797b977bcf5e09cdc" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 37333 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c59f513dd13b45f4a554c64ff7a03c87" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 38723 } health_report { overall_health: HEALTHY } } }
I20260925 07:32:52.358634 23713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:52.356278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37242 (local address 127.9.66.2:38723)
0925 07:32:52.356345 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 07:32:52.356347 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:32:52.356354 (+ 7us) server_negotiation.cc:407] Connection header received
0925 07:32:52.356393 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:32:52.356394 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:32:52.356432 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:32:52.356489 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 07:32:52.356629 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:32:52.357084 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:32:52.357884 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:32:52.358074 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:32:52.358145 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:32:52.358320 (+ 175us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 07:32:52.358446 (+ 126us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 07:32:52.358512 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 07:32:52.358545 (+ 33us) server_negotiation.cc:299] Negotiation successful
0925 07:32:52.358564 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 07:32:52.360410 23656 raft_consensus.cc:1261] T 01813c803dd8444596df2261f1e94ef2 P 74c3871e1dde4f84bda59faeceb249fe [term 1 FOLLOWER]: Refusing update from remote peer c59f513dd13b45f4a554c64ff7a03c87: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:32:52.360476 23388 raft_consensus.cc:1261] T 01813c803dd8444596df2261f1e94ef2 P 0b7b4fae549a422797b977bcf5e09cdc [term 1 FOLLOWER]: Refusing update from remote peer c59f513dd13b45f4a554c64ff7a03c87: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:32:52.360801 23722 consensus_queue.cc:1035] T 01813c803dd8444596df2261f1e94ef2 P c59f513dd13b45f4a554c64ff7a03c87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74c3871e1dde4f84bda59faeceb249fe" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 43023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:32:52.360950 23726 consensus_queue.cc:1035] T 01813c803dd8444596df2261f1e94ef2 P c59f513dd13b45f4a554c64ff7a03c87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b7b4fae549a422797b977bcf5e09cdc" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 37333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:32:52.364745 23729 mvcc.cc:205] Tried to move back new op lower bound from 7333157160385589248 to 7333157160084725760. Current Snapshot: MvccSnapshot[applied={T|T < 7333157160385589248 or (T in {7333157160385589248})}]
I20260925 07:32:52.366317 23243 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37762:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 07:32:52.366428 23243 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37762:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 07:32:52.367193 23243 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 265a353e5527411ca0c035891c95121e: Sending DeleteTablet for 3 replicas of tablet 01813c803dd8444596df2261f1e94ef2
I20260925 07:32:52.367527 23502 tablet_service.cc:1514] Processing DeleteTablet for tablet 01813c803dd8444596df2261f1e94ef2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:32:52 UTC) from {username='kudu', principal='kudu/127.9.66.62@KRBTEST.COM'} at 127.0.0.1:37232
I20260925 07:32:52.367558 23636 tablet_service.cc:1514] Processing DeleteTablet for tablet 01813c803dd8444596df2261f1e94ef2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:32:52 UTC) from {username='kudu', principal='kudu/127.9.66.62@KRBTEST.COM'} at 127.0.0.1:53652
I20260925 07:32:52.367782 23739 tablet_replica.cc:330] T 01813c803dd8444596df2261f1e94ef2 P 74c3871e1dde4f84bda59faeceb249fe: stopping tablet replica
I20260925 07:32:52.367785 23368 tablet_service.cc:1514] Processing DeleteTablet for tablet 01813c803dd8444596df2261f1e94ef2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:32:52 UTC) from {username='kudu', principal='kudu/127.9.66.62@KRBTEST.COM'} at 127.0.0.1:50828
I20260925 07:32:52.367903 23739 raft_consensus.cc:2229] T 01813c803dd8444596df2261f1e94ef2 P 74c3871e1dde4f84bda59faeceb249fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 07:32:52.368021 23740 tablet_replica.cc:330] T 01813c803dd8444596df2261f1e94ef2 P 0b7b4fae549a422797b977bcf5e09cdc: stopping tablet replica
I20260925 07:32:52.368046 23739 raft_consensus.cc:2258] T 01813c803dd8444596df2261f1e94ef2 P 74c3871e1dde4f84bda59faeceb249fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:32:52.368045 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 23305
I20260925 07:32:52.369094 23738 tablet_replica.cc:330] T 01813c803dd8444596df2261f1e94ef2 P c59f513dd13b45f4a554c64ff7a03c87: stopping tablet replica
I20260925 07:32:52.369232 23738 raft_consensus.cc:2229] T 01813c803dd8444596df2261f1e94ef2 P c59f513dd13b45f4a554c64ff7a03c87 [term 1 LEADER]: Raft consensus shutting down.
I20260925 07:32:52.369421 23738 raft_consensus.cc:2258] T 01813c803dd8444596df2261f1e94ef2 P c59f513dd13b45f4a554c64ff7a03c87 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:32:52.369441 23739 ts_tablet_manager.cc:1905] T 01813c803dd8444596df2261f1e94ef2 P 74c3871e1dde4f84bda59faeceb249fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 07:32:52.369946 23738 ts_tablet_manager.cc:1905] T 01813c803dd8444596df2261f1e94ef2 P c59f513dd13b45f4a554c64ff7a03c87: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 07:32:52.371217 23739 ts_tablet_manager.cc:1918] T 01813c803dd8444596df2261f1e94ef2 P 74c3871e1dde4f84bda59faeceb249fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 07:32:52.371346 23739 log.cc:1196] T 01813c803dd8444596df2261f1e94ef2 P 74c3871e1dde4f84bda59faeceb249fe: Deleting WAL directory at /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-2/wal/wals/01813c803dd8444596df2261f1e94ef2
I20260925 07:32:52.371639 23738 ts_tablet_manager.cc:1918] T 01813c803dd8444596df2261f1e94ef2 P c59f513dd13b45f4a554c64ff7a03c87: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 07:32:52.371687 23739 ts_tablet_manager.cc:1939] T 01813c803dd8444596df2261f1e94ef2 P 74c3871e1dde4f84bda59faeceb249fe: Deleting consensus metadata
I20260925 07:32:52.371747 23738 log.cc:1196] T 01813c803dd8444596df2261f1e94ef2 P c59f513dd13b45f4a554c64ff7a03c87: Deleting WAL directory at /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-1/wal/wals/01813c803dd8444596df2261f1e94ef2
I20260925 07:32:52.372001 23738 ts_tablet_manager.cc:1939] T 01813c803dd8444596df2261f1e94ef2 P c59f513dd13b45f4a554c64ff7a03c87: Deleting consensus metadata
I20260925 07:32:52.372213 23228 catalog_manager.cc:4841] TS 74c3871e1dde4f84bda59faeceb249fe (127.9.66.3:43023): tablet 01813c803dd8444596df2261f1e94ef2 (table test-table [id=36c3b7969d1a42cc8f2992aad6751705]) successfully deleted
I20260925 07:32:52.372442 23230 catalog_manager.cc:4841] TS c59f513dd13b45f4a554c64ff7a03c87 (127.9.66.2:38723): tablet 01813c803dd8444596df2261f1e94ef2 (table test-table [id=36c3b7969d1a42cc8f2992aad6751705]) successfully deleted
W20260925 07:32:52.374176 23229 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.66.1:37333 (error 108)
I20260925 07:32:52.374629 23712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:52.374497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.1:37333 (local address 127.0.0.1:50838)
0925 07:32:52.374560 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0925 07:32:52.374599 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.1:37333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260925 07:32:52.374675 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 23439
W20260925 07:32:52.374764 23229 catalog_manager.cc:4584] TS 0b7b4fae549a422797b977bcf5e09cdc (127.9.66.1:37333): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 01813c803dd8444596df2261f1e94ef2: Network error: Client connection negotiation failed: client connection to 127.9.66.1:37333: connect: Connection refused (error 111)
I20260925 07:32:52.379215 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 23573
I20260925 07:32:52.383491 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 23211
I20260925 07:32:52.388230 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.66.62:46577
--webserver_interface=127.9.66.62
--webserver_port=35781
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab
--principal=kudu/127.9.66.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:35859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.66.62:46577
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:32:52.465348 23742 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:32:52.465547 23742 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:32:52.465569 23742 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:32:52.466795 23742 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:32:52.466837 23742 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:32:52.466851 23742 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:32:52.466915 23742 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:32:52.466945 23742 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:32:52.468464 23742 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:35859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.9.66.62:46577
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab
--principal=kudu/127.9.66.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.66.62:46577
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.66.62
--webserver_port=35781
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23742
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:32:52.468713 23742 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:32:52.468940 23742 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:32:52.477196 23749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:32:52.477207 23751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:32:52.477210 23748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:32:52.477231 23742 server_base.cc:1034] running on GCE node
I20260925 07:32:52.477690 23742 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:32:52.478008 23742 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:32:52.479166 23742 hybrid_clock.cc:648] HybridClock initialized: now 1790321572479152 us; error 43 us; skew 500 ppm
Sep 25 07:32:52 dist-test-slave-57fn krb5kdc[23198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321572, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:32:52.481415 23742 init.cc:377] Logged in from keytab as kudu/127.9.66.62@KRBTEST.COM (short username kudu)
I20260925 07:32:52.481882 23742 webserver.cc:460] Webserver started at http://127.9.66.62:35781/ using document root <none> and password file <none>
I20260925 07:32:52.482075 23742 fs_manager.cc:359] Metadata directory not provided
I20260925 07:32:52.482124 23742 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:32:52.483417 23742 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 07:32:52.484014 23758 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:32:52.484202 23742 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 07:32:52.484290 23742 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/master-0/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/master-0/wal
uuid: "265a353e5527411ca0c035891c95121e"
format_stamp: "Formatted at 2026-09-25 07:32:51 on dist-test-slave-57fn"
server_key: "4f72d3af8ddb6c5ae0dd1db167acd81a"
server_key_iv: "eacc4d534c2689883e186a27aea4d983"
server_key_version: "encryptionkey@0"
I20260925 07:32:52.484651 23742 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:32:52.526639 23742 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:32:52.528640 23742 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:32:52.528786 23742 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:32:52.532928 23742 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.62:46577
I20260925 07:32:52.533038 23810 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.62:46577 every 8 connection(s)
I20260925 07:32:52.533396 23742 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
I20260925 07:32:52.535104 23811 sys_catalog.cc:263] Verifying existing consensus state
I20260925 07:32:52.535779 23811 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 265a353e5527411ca0c035891c95121e: Bootstrap starting.
I20260925 07:32:52.543288 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 23742
I20260925 07:32:52.543630 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.66.1:37333
--local_ip_for_outbound_sockets=127.9.66.1
--tserver_master_addrs=127.9.66.62:46577
--webserver_port=46549
--webserver_interface=127.9.66.1
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.1.keytab
--principal=kudu/127.9.66.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:35859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 07:32:52.544399 23811 log.cc:824] T 00000000000000000000000000000000 P 265a353e5527411ca0c035891c95121e: Log is configured to *not* fsync() on all Append() calls
I20260925 07:32:52.546907 23811 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 265a353e5527411ca0c035891c95121e: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 07:32:52.547255 23811 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 265a353e5527411ca0c035891c95121e: Bootstrap complete.
I20260925 07:32:52.551376 23811 raft_consensus.cc:348] T 00000000000000000000000000000000 P 265a353e5527411ca0c035891c95121e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "265a353e5527411ca0c035891c95121e" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 46577 } }
I20260925 07:32:52.551767 23811 raft_consensus.cc:729] T 00000000000000000000000000000000 P 265a353e5527411ca0c035891c95121e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 265a353e5527411ca0c035891c95121e, State: Initialized, Role: FOLLOWER
I20260925 07:32:52.551956 23811 consensus_queue.cc:260] T 00000000000000000000000000000000 P 265a353e5527411ca0c035891c95121e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "265a353e5527411ca0c035891c95121e" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 46577 } }
I20260925 07:32:52.552048 23811 raft_consensus.cc:388] T 00000000000000000000000000000000 P 265a353e5527411ca0c035891c95121e [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 07:32:52.552114 23811 raft_consensus.cc:482] T 00000000000000000000000000000000 P 265a353e5527411ca0c035891c95121e [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 07:32:52.552177 23811 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 265a353e5527411ca0c035891c95121e [term 1 FOLLOWER]: Advancing to term 2
I20260925 07:32:52.553864 23811 raft_consensus.cc:504] T 00000000000000000000000000000000 P 265a353e5527411ca0c035891c95121e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "265a353e5527411ca0c035891c95121e" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 46577 } }
I20260925 07:32:52.553993 23811 leader_election.cc:302] T 00000000000000000000000000000000 P 265a353e5527411ca0c035891c95121e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 265a353e5527411ca0c035891c95121e; no voters:
I20260925 07:32:52.554171 23811 leader_election.cc:288] T 00000000000000000000000000000000 P 265a353e5527411ca0c035891c95121e [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 07:32:52.554234 23815 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 265a353e5527411ca0c035891c95121e [term 2 FOLLOWER]: Leader election won for term 2
I20260925 07:32:52.554378 23815 raft_consensus.cc:686] T 00000000000000000000000000000000 P 265a353e5527411ca0c035891c95121e [term 2 LEADER]: Becoming Leader. State: Replica: 265a353e5527411ca0c035891c95121e, State: Running, Role: LEADER
I20260925 07:32:52.554463 23811 sys_catalog.cc:564] T 00000000000000000000000000000000 P 265a353e5527411ca0c035891c95121e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:32:52.554518 23815 consensus_queue.cc:237] T 00000000000000000000000000000000 P 265a353e5527411ca0c035891c95121e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "265a353e5527411ca0c035891c95121e" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 46577 } }
I20260925 07:32:52.554790 23815 sys_catalog.cc:455] T 00000000000000000000000000000000 P 265a353e5527411ca0c035891c95121e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 265a353e5527411ca0c035891c95121e. Latest consensus state: current_term: 2 leader_uuid: "265a353e5527411ca0c035891c95121e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "265a353e5527411ca0c035891c95121e" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 46577 } } }
I20260925 07:32:52.554790 23817 sys_catalog.cc:455] T 00000000000000000000000000000000 P 265a353e5527411ca0c035891c95121e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "265a353e5527411ca0c035891c95121e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "265a353e5527411ca0c035891c95121e" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 46577 } } }
I20260925 07:32:52.554960 23815 sys_catalog.cc:458] T 00000000000000000000000000000000 P 265a353e5527411ca0c035891c95121e [sys.catalog]: This master's current role is: LEADER
I20260925 07:32:52.554966 23817 sys_catalog.cc:458] T 00000000000000000000000000000000 P 265a353e5527411ca0c035891c95121e [sys.catalog]: This master's current role is: LEADER
I20260925 07:32:52.555184 23821 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 07:32:52.555998 23821 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 07:32:52.556174 23821 catalog_manager.cc:1259] Loaded cluster ID: 0801089512b5455085114a5cfb31516e
I20260925 07:32:52.556393 23821 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 07:32:52.556650 23821 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260925 07:32:52.561210 23821 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260925 07:32:52.563516 23821 catalog_manager.cc:1504] Loading token signing keys...
I20260925 07:32:52.563686 23821 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 265a353e5527411ca0c035891c95121e: Loaded TSK: 0
I20260925 07:32:52.563903 23821 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 07:32:52.629086 23814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:32:52.629309 23814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:32:52.629348 23814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:32:52.630452 23814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:32:52.630520 23814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:32:52.630566 23814 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.66.1
I20260925 07:32:52.631958 23814 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:35859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.1.keytab
--principal=kudu/127.9.66.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.66.1:37333
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.66.1
--webserver_port=46549
--tserver_master_addrs=127.9.66.62:46577
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23814
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.66.1
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:32:52.632225 23814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:32:52.632449 23814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:32:52.638015 23839 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:32:52.638025 23838 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:32:52.638039 23841 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:32:52.638304 23814 server_base.cc:1034] running on GCE node
I20260925 07:32:52.638595 23814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:32:52.638863 23814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:32:52.640058 23814 hybrid_clock.cc:648] HybridClock initialized: now 1790321572640043 us; error 32 us; skew 500 ppm
Sep 25 07:32:52 dist-test-slave-57fn krb5kdc[23198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321572, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:32:52.642586 23814 init.cc:377] Logged in from keytab as kudu/127.9.66.1@KRBTEST.COM (short username kudu)
I20260925 07:32:52.643076 23814 webserver.cc:460] Webserver started at http://127.9.66.1:46549/ using document root <none> and password file <none>
I20260925 07:32:52.643334 23814 fs_manager.cc:359] Metadata directory not provided
I20260925 07:32:52.643414 23814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:32:52.644706 23814 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 07:32:52.645395 23848 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:32:52.645632 23814 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 07:32:52.645716 23814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-0/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-0/wal
uuid: "0b7b4fae549a422797b977bcf5e09cdc"
format_stamp: "Formatted at 2026-09-25 07:32:51 on dist-test-slave-57fn"
server_key: "85dcf2e91ba7b87705987c903cabf23b"
server_key_iv: "52b33f563b20366aa695e29ac5be35ff"
server_key_version: "encryptionkey@0"
I20260925 07:32:52.646034 23814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:32:52.671056 23814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:32:52.673067 23814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:32:52.673205 23814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:32:52.673466 23814 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:32:52.674175 23855 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 07:32:52.675879 23814 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 07:32:52.675952 23814 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260925 07:32:52.675992 23814 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 07:32:52.676558 23814 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 07:32:52.676608 23814 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260925 07:32:52.676684 23855 tablet_bootstrap.cc:492] T 01813c803dd8444596df2261f1e94ef2 P 0b7b4fae549a422797b977bcf5e09cdc: Bootstrap starting.
I20260925 07:32:52.683152 23814 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.1:37333
I20260925 07:32:52.683212 23962 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.1:37333 every 8 connection(s)
I20260925 07:32:52.683525 23814 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-0/data/info.pb
I20260925 07:32:52.686357 23855 log.cc:824] T 01813c803dd8444596df2261f1e94ef2 P 0b7b4fae549a422797b977bcf5e09cdc: Log is configured to *not* fsync() on all Append() calls
I20260925 07:32:52.688452 23855 tablet_bootstrap.cc:492] T 01813c803dd8444596df2261f1e94ef2 P 0b7b4fae549a422797b977bcf5e09cdc: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 07:32:52.688735 23855 tablet_bootstrap.cc:492] T 01813c803dd8444596df2261f1e94ef2 P 0b7b4fae549a422797b977bcf5e09cdc: Bootstrap complete.
I20260925 07:32:52.689291 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 23814
I20260925 07:32:52.689587 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.66.2:38723
--local_ip_for_outbound_sockets=127.9.66.2
--tserver_master_addrs=127.9.66.62:46577
--webserver_port=35775
--webserver_interface=127.9.66.2
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.2.keytab
--principal=kudu/127.9.66.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:35859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 07:32:52.689966 23855 ts_tablet_manager.cc:1397] T 01813c803dd8444596df2261f1e94ef2 P 0b7b4fae549a422797b977bcf5e09cdc: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.013s
Sep 25 07:32:52 dist-test-slave-57fn krb5kdc[23198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321572, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.1@KRBTEST.COM for kudu/127.9.66.62@KRBTEST.COM
I20260925 07:32:52.692232 23855 raft_consensus.cc:348] T 01813c803dd8444596df2261f1e94ef2 P 0b7b4fae549a422797b977bcf5e09cdc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c3871e1dde4f84bda59faeceb249fe" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 43023 } } peers { permanent_uuid: "0b7b4fae549a422797b977bcf5e09cdc" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 37333 } } peers { permanent_uuid: "c59f513dd13b45f4a554c64ff7a03c87" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 38723 } }
I20260925 07:32:52.692576 23855 raft_consensus.cc:729] T 01813c803dd8444596df2261f1e94ef2 P 0b7b4fae549a422797b977bcf5e09cdc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b7b4fae549a422797b977bcf5e09cdc, State: Initialized, Role: FOLLOWER
I20260925 07:32:52.692705 23855 consensus_queue.cc:260] T 01813c803dd8444596df2261f1e94ef2 P 0b7b4fae549a422797b977bcf5e09cdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c3871e1dde4f84bda59faeceb249fe" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 43023 } } peers { permanent_uuid: "0b7b4fae549a422797b977bcf5e09cdc" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 37333 } } peers { permanent_uuid: "c59f513dd13b45f4a554c64ff7a03c87" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 38723 } }
I20260925 07:32:52.692981 23855 ts_tablet_manager.cc:1428] T 01813c803dd8444596df2261f1e94ef2 P 0b7b4fae549a422797b977bcf5e09cdc: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 07:32:52.695431 23965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:52.684198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:46577 (local address 127.9.66.1:36251)
0925 07:32:52.684424 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0925 07:32:52.684457 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 07:32:52.684842 (+ 385us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:32:52.685952 (+ 1110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:32:52.685956 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:32:52.686232 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:32:52.686406 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:32:52.686411 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:32:52.687569 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:32:52.687570 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:32:52.688328 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:32:52.688330 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:32:52.689120 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:32:52.689121 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:32:52.689241 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:32:52.689933 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:32:52.689940 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:32:52.691618 (+ 1678us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:32:52.693969 (+ 2351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:32:52.693974 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:32:52.693976 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:32:52.694181 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:32:52.694480 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:32:52.694481 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:32:52.694482 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:32:52.694600 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:32:52.695026 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:32:52.695030 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:32:52.695123 (+ 93us) client_negotiation.cc:769] Sending connection context
0925 07:32:52.695192 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 07:32:52.695293 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":115,"threads_started":1}
I20260925 07:32:52.695680 23966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:52.684249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.1:36251 (local address 127.9.66.62:46577)
0925 07:32:52.685611 (+ 1362us) server_negotiation.cc:206] Beginning negotiation
0925 07:32:52.685617 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 07:32:52.685642 (+ 25us) server_negotiation.cc:407] Connection header received
0925 07:32:52.685736 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:32:52.685740 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:32:52.685776 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:32:52.685924 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:32:52.686517 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:32:52.687454 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:32:52.688514 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:32:52.688987 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:32:52.689095 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:32:52.691738 (+ 2643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:32:52.691743 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:32:52.691743 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:32:52.691755 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:32:52.693896 (+ 2141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:32:52.694277 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:32:52.694280 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:32:52.694281 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:32:52.694391 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:32:52.694668 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:32:52.694669 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:32:52.694669 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:32:52.694957 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:32:52.695046 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 07:32:52.695277 (+ 231us) server_negotiation.cc:299] Negotiation successful
0925 07:32:52.695443 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1325,"thread_start_us":92,"threads_started":1}
I20260925 07:32:52.696288 23963 heartbeater.cc:347] Connected to a master server at 127.9.66.62:46577
I20260925 07:32:52.696383 23963 heartbeater.cc:464] Registering TS with master...
I20260925 07:32:52.696594 23963 heartbeater.cc:511] Master 127.9.66.62:46577 requested a full tablet report, sending...
I20260925 07:32:52.697222 23775 ts_manager.cc:194] Registered new tserver with Master: 0b7b4fae549a422797b977bcf5e09cdc (127.9.66.1:37333)
I20260925 07:32:52.698308 23775 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.66.1@KRBTEST.COM'} at 127.9.66.1:36251
I20260925 07:32:52.699275 23963 heartbeater.cc:503] Master 127.9.66.62:46577 was elected leader, sending a full tablet report...
Sep 25 07:32:52 dist-test-slave-57fn krb5kdc[23198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321572, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.62@KRBTEST.COM for kudu/127.9.66.1@KRBTEST.COM
I20260925 07:32:52.705685 23970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:52.697778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.1:37333 (local address 127.0.0.1:50844)
0925 07:32:52.698567 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0925 07:32:52.698590 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 07:32:52.698672 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:32:52.698845 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:32:52.698848 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:32:52.699092 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:32:52.699129 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:32:52.699131 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:32:52.700156 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:32:52.700157 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:32:52.700706 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:32:52.700708 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:32:52.701174 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:32:52.701177 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:32:52.701248 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:32:52.701721 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:32:52.701725 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:32:52.702889 (+ 1164us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:32:52.704463 (+ 1574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:32:52.704469 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:32:52.704471 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:32:52.704740 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:32:52.704954 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:32:52.704955 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:32:52.704955 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:32:52.705016 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:32:52.705281 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:32:52.705286 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:32:52.705354 (+ 68us) client_negotiation.cc:769] Sending connection context
0925 07:32:52.705450 (+ 96us) client_negotiation.cc:240] Negotiation successful
0925 07:32:52.705592 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":771,"thread_start_us":81,"threads_started":1}
I20260925 07:32:52.705699 23971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:52.697871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50844 (local address 127.9.66.1:37333)
0925 07:32:52.698033 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 07:32:52.698036 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:32:52.698656 (+ 620us) server_negotiation.cc:407] Connection header received
0925 07:32:52.698767 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:32:52.698769 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:32:52.698784 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:32:52.698844 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:32:52.699208 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:32:52.700075 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:32:52.700804 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:32:52.701090 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:32:52.701163 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:32:52.702975 (+ 1812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:32:52.702982 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:32:52.702982 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:32:52.702989 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:32:52.704353 (+ 1364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:32:52.704817 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:32:52.704820 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:32:52.704820 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:32:52.704880 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:32:52.705074 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:32:52.705075 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:32:52.705076 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:32:52.705206 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:32:52.705286 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 07:32:52.705486 (+ 200us) server_negotiation.cc:299] Negotiation successful
0925 07:32:52.705618 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":98,"threads_started":1}
I20260925 07:32:52.706034 23897 tablet_service.cc:1514] Processing DeleteTablet for tablet 01813c803dd8444596df2261f1e94ef2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:32:52 UTC) from {username='kudu', principal='kudu/127.9.66.62@KRBTEST.COM'} at 127.0.0.1:50844
I20260925 07:32:52.706466 23972 tablet_replica.cc:330] stopping tablet replica
I20260925 07:32:52.706566 23972 raft_consensus.cc:2229] T 01813c803dd8444596df2261f1e94ef2 P 0b7b4fae549a422797b977bcf5e09cdc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 07:32:52.706624 23972 raft_consensus.cc:2258] T 01813c803dd8444596df2261f1e94ef2 P 0b7b4fae549a422797b977bcf5e09cdc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:32:52.706912 23972 ts_tablet_manager.cc:1905] T 01813c803dd8444596df2261f1e94ef2 P 0b7b4fae549a422797b977bcf5e09cdc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 07:32:52.709170 23972 ts_tablet_manager.cc:1918] T 01813c803dd8444596df2261f1e94ef2 P 0b7b4fae549a422797b977bcf5e09cdc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 07:32:52.709241 23972 log.cc:1196] T 01813c803dd8444596df2261f1e94ef2 P 0b7b4fae549a422797b977bcf5e09cdc: Deleting WAL directory at /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-0/wal/wals/01813c803dd8444596df2261f1e94ef2
I20260925 07:32:52.709486 23972 ts_tablet_manager.cc:1939] T 01813c803dd8444596df2261f1e94ef2 P 0b7b4fae549a422797b977bcf5e09cdc: Deleting consensus metadata
I20260925 07:32:52.710006 23761 catalog_manager.cc:4841] TS 0b7b4fae549a422797b977bcf5e09cdc (127.9.66.1:37333): tablet 01813c803dd8444596df2261f1e94ef2 (table test-table [id=36c3b7969d1a42cc8f2992aad6751705]) successfully deleted
W20260925 07:32:52.771291 23968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:32:52.771502 23968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:32:52.771528 23968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:32:52.772688 23968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:32:52.772730 23968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:32:52.772758 23968 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.66.2
I20260925 07:32:52.774232 23968 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:35859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.2.keytab
--principal=kudu/127.9.66.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.66.2:38723
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.66.2
--webserver_port=35775
--tserver_master_addrs=127.9.66.62:46577
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23968
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.66.2
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:32:52.774466 23968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:32:52.774701 23968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:32:52.779559 23980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:32:52.779572 23977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:32:52.779578 23978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:32:52.779851 23968 server_base.cc:1034] running on GCE node
I20260925 07:32:52.780079 23968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:32:52.780351 23968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:32:52.781544 23968 hybrid_clock.cc:648] HybridClock initialized: now 1790321572781533 us; error 43 us; skew 500 ppm
Sep 25 07:32:52 dist-test-slave-57fn krb5kdc[23198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321572, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:32:52.784113 23968 init.cc:377] Logged in from keytab as kudu/127.9.66.2@KRBTEST.COM (short username kudu)
I20260925 07:32:52.784581 23968 webserver.cc:460] Webserver started at http://127.9.66.2:35775/ using document root <none> and password file <none>
I20260925 07:32:52.784801 23968 fs_manager.cc:359] Metadata directory not provided
I20260925 07:32:52.784881 23968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:32:52.786278 23968 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 07:32:52.786911 23987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:32:52.787077 23968 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 07:32:52.787207 23968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-1/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-1/wal
uuid: "c59f513dd13b45f4a554c64ff7a03c87"
format_stamp: "Formatted at 2026-09-25 07:32:51 on dist-test-slave-57fn"
server_key: "f2804f27d213bc1967c47cb055e357d5"
server_key_iv: "0cc2b036bc76740d6e03c407966f7a62"
server_key_version: "encryptionkey@0"
I20260925 07:32:52.787581 23968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:32:52.794479 23968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:32:52.796169 23968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:32:52.796284 23968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:32:52.796463 23968 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:32:52.796686 23968 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:32:52.796722 23968 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:32:52.796744 23968 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:32:52.796758 23968 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:32:52.802497 23968 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.2:38723
I20260925 07:32:52.802577 24100 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.2:38723 every 8 connection(s)
I20260925 07:32:52.802790 23968 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-1/data/info.pb
I20260925 07:32:52.805949 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 23968
I20260925 07:32:52.806238 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.66.3:43023
--local_ip_for_outbound_sockets=127.9.66.3
--tserver_master_addrs=127.9.66.62:46577
--webserver_port=45283
--webserver_interface=127.9.66.3
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.3.keytab
--principal=kudu/127.9.66.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:35859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 07:32:52 dist-test-slave-57fn krb5kdc[23198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321572, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.2@KRBTEST.COM for kudu/127.9.66.62@KRBTEST.COM
I20260925 07:32:52.812064 23966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:52.803454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.2:46807 (local address 127.9.66.62:46577)
0925 07:32:52.803559 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0925 07:32:52.803561 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:32:52.803719 (+ 158us) server_negotiation.cc:407] Connection header received
0925 07:32:52.804233 (+ 514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:32:52.804234 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:32:52.804249 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:32:52.804310 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:32:52.805171 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:32:52.805731 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:32:52.806900 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:32:52.807120 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:32:52.807265 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:32:52.809246 (+ 1981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:32:52.809247 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:32:52.809247 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:32:52.809255 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:32:52.810715 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:32:52.811140 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:32:52.811141 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:32:52.811141 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:32:52.811196 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:32:52.811456 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:32:52.811457 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:32:52.811457 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:32:52.811563 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:32:52.811635 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 07:32:52.811894 (+ 259us) server_negotiation.cc:299] Negotiation successful
0925 07:32:52.811979 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 07:32:52.812219 24103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:52.803390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:46577 (local address 127.9.66.2:46807)
0925 07:32:52.803590 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0925 07:32:52.803630 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 07:32:52.804124 (+ 494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:32:52.804380 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:32:52.804383 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:32:52.804907 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:32:52.805074 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:32:52.805078 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:32:52.805829 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:32:52.805830 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:32:52.806802 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:32:52.806804 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:32:52.807272 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:32:52.807273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:32:52.807402 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:32:52.807867 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:32:52.807876 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:32:52.809151 (+ 1275us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:32:52.810809 (+ 1658us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:32:52.810816 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:32:52.810818 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:32:52.811029 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:32:52.811273 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:32:52.811273 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:32:52.811274 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:32:52.811388 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:32:52.811628 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:32:52.811634 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:32:52.811764 (+ 130us) client_negotiation.cc:769] Sending connection context
0925 07:32:52.811866 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 07:32:52.812007 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":89,"threads_started":1}
I20260925 07:32:52.812680 24101 heartbeater.cc:347] Connected to a master server at 127.9.66.62:46577
I20260925 07:32:52.812765 24101 heartbeater.cc:464] Registering TS with master...
I20260925 07:32:52.812951 24101 heartbeater.cc:511] Master 127.9.66.62:46577 requested a full tablet report, sending...
I20260925 07:32:52.813349 23775 ts_manager.cc:194] Registered new tserver with Master: c59f513dd13b45f4a554c64ff7a03c87 (127.9.66.2:38723)
I20260925 07:32:52.813943 23775 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.66.2@KRBTEST.COM'} at 127.9.66.2:46807
W20260925 07:32:52.887151 24104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:32:52.887392 24104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:32:52.887423 24104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:32:52.888612 24104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:32:52.888650 24104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:32:52.888676 24104 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.66.3
I20260925 07:32:52.890198 24104 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:35859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.3.keytab
--principal=kudu/127.9.66.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.66.3:43023
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.66.3
--webserver_port=45283
--tserver_master_addrs=127.9.66.62:46577
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24104
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.66.3
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:32:52.890388 24104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:32:52.890607 24104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 07:32:52.895587 24104 server_base.cc:1034] running on GCE node
W20260925 07:32:52.895560 24109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:32:52.895553 24110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:32:52.895566 24112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:32:52.896119 24104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:32:52.896380 24104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:32:52.897653 24104 hybrid_clock.cc:648] HybridClock initialized: now 1790321572897637 us; error 25 us; skew 500 ppm
Sep 25 07:32:52 dist-test-slave-57fn krb5kdc[23198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321572, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:32:52.900192 24104 init.cc:377] Logged in from keytab as kudu/127.9.66.3@KRBTEST.COM (short username kudu)
I20260925 07:32:52.900691 24104 webserver.cc:460] Webserver started at http://127.9.66.3:45283/ using document root <none> and password file <none>
I20260925 07:32:52.900913 24104 fs_manager.cc:359] Metadata directory not provided
I20260925 07:32:52.900965 24104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:32:52.902360 24104 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:32:52.902964 24119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:32:52.903158 24104 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:32:52.903242 24104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-2/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-2/wal
uuid: "74c3871e1dde4f84bda59faeceb249fe"
format_stamp: "Formatted at 2026-09-25 07:32:51 on dist-test-slave-57fn"
server_key: "f3df166509c3aac269bd51f1d78d232d"
server_key_iv: "4cccb95ed696f0c469ea187ac84a5f5a"
server_key_version: "encryptionkey@0"
I20260925 07:32:52.903573 24104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:32:52.915441 24104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:32:52.917191 24104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:32:52.917312 24104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:32:52.917487 24104 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:32:52.917829 24104 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:32:52.917860 24104 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:32:52.917882 24104 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:32:52.917894 24104 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:32:52.923547 24104 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.3:43023
I20260925 07:32:52.923621 24232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.3:43023 every 8 connection(s)
I20260925 07:32:52.923951 24104 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-2/data/info.pb
Sep 25 07:32:52 dist-test-slave-57fn krb5kdc[23198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321572, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.3@KRBTEST.COM for kudu/127.9.66.62@KRBTEST.COM
I20260925 07:32:52.931891 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 24104
I20260925 07:32:52.933279 23966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:52.924655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.3:59719 (local address 127.9.66.62:46577)
0925 07:32:52.924741 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0925 07:32:52.924743 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:32:52.924899 (+ 156us) server_negotiation.cc:407] Connection header received
0925 07:32:52.925351 (+ 452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:32:52.925352 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:32:52.925362 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:32:52.925419 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:32:52.926415 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:32:52.926909 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:32:52.927853 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:32:52.928029 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:32:52.928079 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:32:52.930257 (+ 2178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:32:52.930258 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:32:52.930259 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:32:52.930266 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:32:52.931612 (+ 1346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:32:52.931971 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:32:52.931973 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:32:52.931974 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:32:52.932041 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:32:52.932589 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:32:52.932590 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:32:52.932590 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:32:52.932718 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:32:52.932769 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 07:32:52.933113 (+ 344us) server_negotiation.cc:299] Negotiation successful
0925 07:32:52.933175 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 07:32:52.933568 24235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:52.924627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:46577 (local address 127.9.66.3:59719)
0925 07:32:52.924784 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 07:32:52.924817 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 07:32:52.925270 (+ 453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:32:52.925468 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:32:52.925471 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:32:52.926114 (+ 643us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:32:52.926337 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:32:52.926342 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:32:52.926994 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:32:52.926995 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:32:52.927779 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:32:52.927781 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:32:52.928099 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:32:52.928099 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:32:52.928197 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:32:52.928732 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:32:52.928740 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:32:52.930154 (+ 1414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:32:52.931681 (+ 1527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:32:52.931686 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:32:52.931687 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:32:52.931890 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:32:52.932382 (+ 492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:32:52.932383 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:32:52.932383 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:32:52.932522 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:32:52.932782 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:32:52.932788 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:32:52.932953 (+ 165us) client_negotiation.cc:769] Sending connection context
0925 07:32:52.933086 (+ 133us) client_negotiation.cc:240] Negotiation successful
0925 07:32:52.933248 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":99,"threads_started":1}
I20260925 07:32:52.934162 24233 heartbeater.cc:347] Connected to a master server at 127.9.66.62:46577
I20260925 07:32:52.934258 24233 heartbeater.cc:464] Registering TS with master...
I20260925 07:32:52.934432 24233 heartbeater.cc:511] Master 127.9.66.62:46577 requested a full tablet report, sending...
I20260925 07:32:52.934814 23775 ts_manager.cc:194] Registered new tserver with Master: 74c3871e1dde4f84bda59faeceb249fe (127.9.66.3:43023)
I20260925 07:32:52.935247 23775 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.66.3@KRBTEST.COM'} at 127.9.66.3:59719
I20260925 07:32:52.938467 24237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:52.932282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37776 (local address 127.9.66.62:46577)
0925 07:32:52.932397 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0925 07:32:52.932400 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:32:52.932416 (+ 16us) server_negotiation.cc:407] Connection header received
0925 07:32:52.932505 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:32:52.932505 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:32:52.932513 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:32:52.932606 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:32:52.933235 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:32:52.933903 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:32:52.934848 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:32:52.935116 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:32:52.935198 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:32:52.936438 (+ 1240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:32:52.936439 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:32:52.936439 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:32:52.936445 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:32:52.937666 (+ 1221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:32:52.937950 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:32:52.937950 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:32:52.937951 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:32:52.937999 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:32:52.938111 (+ 112us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:32:52.938112 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:32:52.938112 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:32:52.938191 (+ 79us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:32:52.938297 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 07:32:52.938345 (+ 48us) server_negotiation.cc:299] Negotiation successful
0925 07:32:52.938401 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":77,"threads_started":1}
I20260925 07:32:52.939188 9480 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 07:32:52.941618 24237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:52.939494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37788 (local address 127.9.66.62:46577)
0925 07:32:52.939526 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0925 07:32:52.939527 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 07:32:52.939617 (+ 90us) server_negotiation.cc:407] Connection header received
0925 07:32:52.939677 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:32:52.939677 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:32:52.939702 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:32:52.939766 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 07:32:52.939900 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:32:52.940336 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:32:52.940984 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:32:52.941157 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:32:52.941203 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:32:52.941320 (+ 117us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 07:32:52.941411 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 07:32:52.941465 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 07:32:52.941541 (+ 76us) server_negotiation.cc:299] Negotiation successful
0925 07:32:52.941563 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 07:32:52.941856 23775 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:37788:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 07:32:52.943403 23775 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 07:32:52.945308 23897 tablet_service.cc:1467] Processing CreateTablet for tablet ab88041be7cd4c69aa6c46a5890553c9 (DEFAULT_TABLE table=test-table [id=6388dc24096d4539bd3fd161abc852b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:32:52.945500 23897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab88041be7cd4c69aa6c46a5890553c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:32:52.947130 23855 tablet_bootstrap.cc:492] T ab88041be7cd4c69aa6c46a5890553c9 P 0b7b4fae549a422797b977bcf5e09cdc: Bootstrap starting.
I20260925 07:32:52.947430 23855 tablet_bootstrap.cc:654] T ab88041be7cd4c69aa6c46a5890553c9 P 0b7b4fae549a422797b977bcf5e09cdc: Neither blocks nor log segments found. Creating new log.
I20260925 07:32:52.948280 23855 tablet_bootstrap.cc:492] T ab88041be7cd4c69aa6c46a5890553c9 P 0b7b4fae549a422797b977bcf5e09cdc: No bootstrap required, opened a new log
I20260925 07:32:52.948356 23855 ts_tablet_manager.cc:1397] T ab88041be7cd4c69aa6c46a5890553c9 P 0b7b4fae549a422797b977bcf5e09cdc: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260925 07:32:52.948514 23855 raft_consensus.cc:348] T ab88041be7cd4c69aa6c46a5890553c9 P 0b7b4fae549a422797b977bcf5e09cdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c59f513dd13b45f4a554c64ff7a03c87" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 38723 } } peers { permanent_uuid: "74c3871e1dde4f84bda59faeceb249fe" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 43023 } } peers { permanent_uuid: "0b7b4fae549a422797b977bcf5e09cdc" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 37333 } }
I20260925 07:32:52.948590 23855 raft_consensus.cc:374] T ab88041be7cd4c69aa6c46a5890553c9 P 0b7b4fae549a422797b977bcf5e09cdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:32:52.948638 23855 raft_consensus.cc:729] T ab88041be7cd4c69aa6c46a5890553c9 P 0b7b4fae549a422797b977bcf5e09cdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b7b4fae549a422797b977bcf5e09cdc, State: Initialized, Role: FOLLOWER
I20260925 07:32:52.948678 23855 consensus_queue.cc:260] T ab88041be7cd4c69aa6c46a5890553c9 P 0b7b4fae549a422797b977bcf5e09cdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c59f513dd13b45f4a554c64ff7a03c87" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 38723 } } peers { permanent_uuid: "74c3871e1dde4f84bda59faeceb249fe" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 43023 } } peers { permanent_uuid: "0b7b4fae549a422797b977bcf5e09cdc" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 37333 } }
I20260925 07:32:52.948774 23855 ts_tablet_manager.cc:1428] T ab88041be7cd4c69aa6c46a5890553c9 P 0b7b4fae549a422797b977bcf5e09cdc: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 07:32:52.949447 23970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:52.945087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.2:38723 (local address 127.0.0.1:37250)
0925 07:32:52.945121 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0925 07:32:52.945132 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 07:32:52.945166 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:32:52.945732 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:32:52.945733 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:32:52.945736 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:32:52.945775 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:32:52.945776 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:32:52.946943 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:32:52.946944 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:32:52.948203 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:32:52.948206 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:32:52.949181 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:32:52.949182 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:32:52.949255 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:32:52.949275 (+ 20us) client_negotiation.cc:769] Sending connection context
0925 07:32:52.949330 (+ 55us) client_negotiation.cc:240] Negotiation successful
0925 07:32:52.949378 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20,"mutex_wait_us":41}
I20260925 07:32:52.949479 24241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:52.945310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37250 (local address 127.9.66.2:38723)
0925 07:32:52.945546 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0925 07:32:52.945551 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 07:32:52.945562 (+ 11us) server_negotiation.cc:407] Connection header received
0925 07:32:52.945605 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:32:52.945607 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:32:52.945624 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:32:52.945716 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:32:52.945834 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:32:52.946808 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:32:52.948291 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:32:52.949031 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:32:52.949093 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:32:52.949148 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 07:32:52.949349 (+ 201us) server_negotiation.cc:299] Negotiation successful
0925 07:32:52.949401 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":87,"threads_started":1}
I20260925 07:32:52.949682 24240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:52.945310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53664 (local address 127.9.66.3:43023)
0925 07:32:52.945573 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0925 07:32:52.945578 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 07:32:52.945729 (+ 151us) server_negotiation.cc:407] Connection header received
0925 07:32:52.945764 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:32:52.945766 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:32:52.945779 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:32:52.945846 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:32:52.946022 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:32:52.946833 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:32:52.948375 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:32:52.949244 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:32:52.949305 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:32:52.949356 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 07:32:52.949565 (+ 209us) server_negotiation.cc:299] Negotiation successful
0925 07:32:52.949610 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":89,"threads_started":1}
I20260925 07:32:52.949906 24239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:52.945096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.3:43023 (local address 127.0.0.1:53664)
0925 07:32:52.945599 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0925 07:32:52.945609 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 07:32:52.945676 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:32:52.945909 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:32:52.945909 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:32:52.945911 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:32:52.945937 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:32:52.945938 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:32:52.946951 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:32:52.946952 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:32:52.948297 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:32:52.948299 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:32:52.949386 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:32:52.949387 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:32:52.949445 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:32:52.949448 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 07:32:52.949792 (+ 344us) client_negotiation.cc:240] Negotiation successful
0925 07:32:52.949836 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"mutex_wait_us":59,"thread_start_us":93,"threads_started":1}
I20260925 07:32:52.950390 24035 tablet_service.cc:1467] Processing CreateTablet for tablet ab88041be7cd4c69aa6c46a5890553c9 (DEFAULT_TABLE table=test-table [id=6388dc24096d4539bd3fd161abc852b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:32:52.950512 24167 tablet_service.cc:1467] Processing CreateTablet for tablet ab88041be7cd4c69aa6c46a5890553c9 (DEFAULT_TABLE table=test-table [id=6388dc24096d4539bd3fd161abc852b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:32:52.950747 24035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab88041be7cd4c69aa6c46a5890553c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:32:52.950774 24167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab88041be7cd4c69aa6c46a5890553c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:32:52.954219 24243 tablet_bootstrap.cc:492] T ab88041be7cd4c69aa6c46a5890553c9 P c59f513dd13b45f4a554c64ff7a03c87: Bootstrap starting.
I20260925 07:32:52.954763 24243 tablet_bootstrap.cc:654] T ab88041be7cd4c69aa6c46a5890553c9 P c59f513dd13b45f4a554c64ff7a03c87: Neither blocks nor log segments found. Creating new log.
I20260925 07:32:52.955320 24243 log.cc:824] T ab88041be7cd4c69aa6c46a5890553c9 P c59f513dd13b45f4a554c64ff7a03c87: Log is configured to *not* fsync() on all Append() calls
I20260925 07:32:52.955574 24244 tablet_bootstrap.cc:492] T ab88041be7cd4c69aa6c46a5890553c9 P 74c3871e1dde4f84bda59faeceb249fe: Bootstrap starting.
I20260925 07:32:52.956007 24243 tablet_bootstrap.cc:492] T ab88041be7cd4c69aa6c46a5890553c9 P c59f513dd13b45f4a554c64ff7a03c87: No bootstrap required, opened a new log
I20260925 07:32:52.956154 24244 tablet_bootstrap.cc:654] T ab88041be7cd4c69aa6c46a5890553c9 P 74c3871e1dde4f84bda59faeceb249fe: Neither blocks nor log segments found. Creating new log.
I20260925 07:32:52.956159 24243 ts_tablet_manager.cc:1397] T ab88041be7cd4c69aa6c46a5890553c9 P c59f513dd13b45f4a554c64ff7a03c87: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 07:32:52.956674 24244 log.cc:824] T ab88041be7cd4c69aa6c46a5890553c9 P 74c3871e1dde4f84bda59faeceb249fe: Log is configured to *not* fsync() on all Append() calls
I20260925 07:32:52.957319 24244 tablet_bootstrap.cc:492] T ab88041be7cd4c69aa6c46a5890553c9 P 74c3871e1dde4f84bda59faeceb249fe: No bootstrap required, opened a new log
I20260925 07:32:52.957394 24244 ts_tablet_manager.cc:1397] T ab88041be7cd4c69aa6c46a5890553c9 P 74c3871e1dde4f84bda59faeceb249fe: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 07:32:52.957772 24243 raft_consensus.cc:348] T ab88041be7cd4c69aa6c46a5890553c9 P c59f513dd13b45f4a554c64ff7a03c87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c59f513dd13b45f4a554c64ff7a03c87" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 38723 } } peers { permanent_uuid: "74c3871e1dde4f84bda59faeceb249fe" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 43023 } } peers { permanent_uuid: "0b7b4fae549a422797b977bcf5e09cdc" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 37333 } }
I20260925 07:32:52.957906 24243 raft_consensus.cc:374] T ab88041be7cd4c69aa6c46a5890553c9 P c59f513dd13b45f4a554c64ff7a03c87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:32:52.957926 24243 raft_consensus.cc:729] T ab88041be7cd4c69aa6c46a5890553c9 P c59f513dd13b45f4a554c64ff7a03c87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c59f513dd13b45f4a554c64ff7a03c87, State: Initialized, Role: FOLLOWER
I20260925 07:32:52.958055 24243 consensus_queue.cc:260] T ab88041be7cd4c69aa6c46a5890553c9 P c59f513dd13b45f4a554c64ff7a03c87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c59f513dd13b45f4a554c64ff7a03c87" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 38723 } } peers { permanent_uuid: "74c3871e1dde4f84bda59faeceb249fe" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 43023 } } peers { permanent_uuid: "0b7b4fae549a422797b977bcf5e09cdc" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 37333 } }
I20260925 07:32:52.958359 24243 ts_tablet_manager.cc:1428] T ab88041be7cd4c69aa6c46a5890553c9 P c59f513dd13b45f4a554c64ff7a03c87: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 07:32:52.958382 24101 heartbeater.cc:503] Master 127.9.66.62:46577 was elected leader, sending a full tablet report...
I20260925 07:32:52.959136 24244 raft_consensus.cc:348] T ab88041be7cd4c69aa6c46a5890553c9 P 74c3871e1dde4f84bda59faeceb249fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c59f513dd13b45f4a554c64ff7a03c87" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 38723 } } peers { permanent_uuid: "74c3871e1dde4f84bda59faeceb249fe" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 43023 } } peers { permanent_uuid: "0b7b4fae549a422797b977bcf5e09cdc" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 37333 } }
I20260925 07:32:52.959319 24244 raft_consensus.cc:374] T ab88041be7cd4c69aa6c46a5890553c9 P 74c3871e1dde4f84bda59faeceb249fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:32:52.959371 24244 raft_consensus.cc:729] T ab88041be7cd4c69aa6c46a5890553c9 P 74c3871e1dde4f84bda59faeceb249fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74c3871e1dde4f84bda59faeceb249fe, State: Initialized, Role: FOLLOWER
I20260925 07:32:52.959493 24244 consensus_queue.cc:260] T ab88041be7cd4c69aa6c46a5890553c9 P 74c3871e1dde4f84bda59faeceb249fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c59f513dd13b45f4a554c64ff7a03c87" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 38723 } } peers { permanent_uuid: "74c3871e1dde4f84bda59faeceb249fe" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 43023 } } peers { permanent_uuid: "0b7b4fae549a422797b977bcf5e09cdc" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 37333 } }
I20260925 07:32:52.959758 24244 ts_tablet_manager.cc:1428] T ab88041be7cd4c69aa6c46a5890553c9 P 74c3871e1dde4f84bda59faeceb249fe: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 07:32:52.959785 24233 heartbeater.cc:503] Master 127.9.66.62:46577 was elected leader, sending a full tablet report...
W20260925 07:32:53.053344 24102 tablet.cc:2369] T ab88041be7cd4c69aa6c46a5890553c9 P c59f513dd13b45f4a554c64ff7a03c87: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 07:32:53.055454 23969 raft_consensus.cc:482] T ab88041be7cd4c69aa6c46a5890553c9 P 0b7b4fae549a422797b977bcf5e09cdc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 07:32:53.055575 23969 raft_consensus.cc:504] T ab88041be7cd4c69aa6c46a5890553c9 P 0b7b4fae549a422797b977bcf5e09cdc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c59f513dd13b45f4a554c64ff7a03c87" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 38723 } } peers { permanent_uuid: "74c3871e1dde4f84bda59faeceb249fe" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 43023 } } peers { permanent_uuid: "0b7b4fae549a422797b977bcf5e09cdc" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 37333 } }
I20260925 07:32:53.055955 23969 leader_election.cc:288] T ab88041be7cd4c69aa6c46a5890553c9 P 0b7b4fae549a422797b977bcf5e09cdc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c59f513dd13b45f4a554c64ff7a03c87 (127.9.66.2:38723), 74c3871e1dde4f84bda59faeceb249fe (127.9.66.3:43023)
I20260925 07:32:53.058997 24241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:53.056293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.1:42077 (local address 127.9.66.2:38723)
0925 07:32:53.056350 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 07:32:53.056352 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:32:53.056362 (+ 10us) server_negotiation.cc:407] Connection header received
0925 07:32:53.056394 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:32:53.056395 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:32:53.056407 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:32:53.056478 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:32:53.056671 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:32:53.057147 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:32:53.058206 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:32:53.058595 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:32:53.058682 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:32:53.058692 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 07:32:53.058881 (+ 189us) server_negotiation.cc:299] Negotiation successful
0925 07:32:53.058923 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 07:32:53.059017 23965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:53.056164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.2:38723 (local address 127.9.66.1:42077)
0925 07:32:53.056228 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0925 07:32:53.056242 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 07:32:53.056320 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:32:53.056497 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:32:53.056498 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:32:53.056499 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:32:53.056571 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:32:53.056573 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:32:53.057241 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:32:53.057242 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:32:53.058099 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:32:53.058102 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:32:53.058680 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:32:53.058681 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:32:53.058755 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:32:53.058774 (+ 19us) client_negotiation.cc:769] Sending connection context
0925 07:32:53.058875 (+ 101us) client_negotiation.cc:240] Negotiation successful
0925 07:32:53.058928 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260925 07:32:53.059303 24249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:53.056175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.3:43023 (local address 127.9.66.1:42505)
0925 07:32:53.056667 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0925 07:32:53.056677 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 07:32:53.056729 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:32:53.056868 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:32:53.056869 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:32:53.056870 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:32:53.056949 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:32:53.056951 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:32:53.057618 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:32:53.057619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:32:53.058443 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:32:53.058446 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:32:53.059028 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:32:53.059029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:32:53.059097 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:32:53.059102 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 07:32:53.059165 (+ 63us) client_negotiation.cc:240] Negotiation successful
0925 07:32:53.059208 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"mutex_wait_us":334,"thread_start_us":82,"threads_started":1}
I20260925 07:32:53.059304 24240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:53.056253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.1:42505 (local address 127.9.66.3:43023)
0925 07:32:53.056309 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 07:32:53.056311 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:32:53.056722 (+ 411us) server_negotiation.cc:407] Connection header received
0925 07:32:53.056782 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:32:53.056783 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:32:53.056796 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:32:53.056866 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:32:53.057041 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:32:53.057500 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:32:53.058549 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:32:53.058938 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:32:53.059027 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:32:53.059036 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 07:32:53.059187 (+ 151us) server_negotiation.cc:299] Negotiation successful
0925 07:32:53.059236 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 07:32:53.059396 24055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab88041be7cd4c69aa6c46a5890553c9" candidate_uuid: "0b7b4fae549a422797b977bcf5e09cdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c59f513dd13b45f4a554c64ff7a03c87" is_pre_election: true
I20260925 07:32:53.059587 24055 raft_consensus.cc:2454] T ab88041be7cd4c69aa6c46a5890553c9 P c59f513dd13b45f4a554c64ff7a03c87 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b7b4fae549a422797b977bcf5e09cdc in term 0.
I20260925 07:32:53.059767 24187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab88041be7cd4c69aa6c46a5890553c9" candidate_uuid: "0b7b4fae549a422797b977bcf5e09cdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74c3871e1dde4f84bda59faeceb249fe" is_pre_election: true
I20260925 07:32:53.059828 23852 leader_election.cc:302] T ab88041be7cd4c69aa6c46a5890553c9 P 0b7b4fae549a422797b977bcf5e09cdc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b7b4fae549a422797b977bcf5e09cdc, c59f513dd13b45f4a554c64ff7a03c87; no voters:
I20260925 07:32:53.059926 24187 raft_consensus.cc:2454] T ab88041be7cd4c69aa6c46a5890553c9 P 74c3871e1dde4f84bda59faeceb249fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b7b4fae549a422797b977bcf5e09cdc in term 0.
I20260925 07:32:53.059942 23969 raft_consensus.cc:2781] T ab88041be7cd4c69aa6c46a5890553c9 P 0b7b4fae549a422797b977bcf5e09cdc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 07:32:53.060019 23969 raft_consensus.cc:482] T ab88041be7cd4c69aa6c46a5890553c9 P 0b7b4fae549a422797b977bcf5e09cdc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 07:32:53.060045 23969 raft_consensus.cc:3037] T ab88041be7cd4c69aa6c46a5890553c9 P 0b7b4fae549a422797b977bcf5e09cdc [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:32:53.060755 23969 raft_consensus.cc:504] T ab88041be7cd4c69aa6c46a5890553c9 P 0b7b4fae549a422797b977bcf5e09cdc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c59f513dd13b45f4a554c64ff7a03c87" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 38723 } } peers { permanent_uuid: "74c3871e1dde4f84bda59faeceb249fe" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 43023 } } peers { permanent_uuid: "0b7b4fae549a422797b977bcf5e09cdc" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 37333 } }
I20260925 07:32:53.060899 23969 leader_election.cc:288] T ab88041be7cd4c69aa6c46a5890553c9 P 0b7b4fae549a422797b977bcf5e09cdc [CANDIDATE]: Term 1 election: Requested vote from peers c59f513dd13b45f4a554c64ff7a03c87 (127.9.66.2:38723), 74c3871e1dde4f84bda59faeceb249fe (127.9.66.3:43023)
I20260925 07:32:53.061077 24187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab88041be7cd4c69aa6c46a5890553c9" candidate_uuid: "0b7b4fae549a422797b977bcf5e09cdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74c3871e1dde4f84bda59faeceb249fe"
I20260925 07:32:53.061077 24055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab88041be7cd4c69aa6c46a5890553c9" candidate_uuid: "0b7b4fae549a422797b977bcf5e09cdc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c59f513dd13b45f4a554c64ff7a03c87"
I20260925 07:32:53.061156 24187 raft_consensus.cc:3037] T ab88041be7cd4c69aa6c46a5890553c9 P 74c3871e1dde4f84bda59faeceb249fe [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:32:53.061156 24055 raft_consensus.cc:3037] T ab88041be7cd4c69aa6c46a5890553c9 P c59f513dd13b45f4a554c64ff7a03c87 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:32:53.062081 24055 raft_consensus.cc:2454] T ab88041be7cd4c69aa6c46a5890553c9 P c59f513dd13b45f4a554c64ff7a03c87 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b7b4fae549a422797b977bcf5e09cdc in term 1.
I20260925 07:32:53.062078 24187 raft_consensus.cc:2454] T ab88041be7cd4c69aa6c46a5890553c9 P 74c3871e1dde4f84bda59faeceb249fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b7b4fae549a422797b977bcf5e09cdc in term 1.
I20260925 07:32:53.062314 23850 leader_election.cc:302] T ab88041be7cd4c69aa6c46a5890553c9 P 0b7b4fae549a422797b977bcf5e09cdc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b7b4fae549a422797b977bcf5e09cdc, 74c3871e1dde4f84bda59faeceb249fe; no voters:
I20260925 07:32:53.062423 23969 raft_consensus.cc:2781] T ab88041be7cd4c69aa6c46a5890553c9 P 0b7b4fae549a422797b977bcf5e09cdc [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:32:53.062590 23969 raft_consensus.cc:686] T ab88041be7cd4c69aa6c46a5890553c9 P 0b7b4fae549a422797b977bcf5e09cdc [term 1 LEADER]: Becoming Leader. State: Replica: 0b7b4fae549a422797b977bcf5e09cdc, State: Running, Role: LEADER
I20260925 07:32:53.062675 23969 consensus_queue.cc:237] T ab88041be7cd4c69aa6c46a5890553c9 P 0b7b4fae549a422797b977bcf5e09cdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c59f513dd13b45f4a554c64ff7a03c87" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 38723 } } peers { permanent_uuid: "74c3871e1dde4f84bda59faeceb249fe" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 43023 } } peers { permanent_uuid: "0b7b4fae549a422797b977bcf5e09cdc" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 37333 } }
I20260925 07:32:53.063462 23774 catalog_manager.cc:5495] T ab88041be7cd4c69aa6c46a5890553c9 P 0b7b4fae549a422797b977bcf5e09cdc reported cstate change: term changed from 0 to 1, leader changed from <none> to 0b7b4fae549a422797b977bcf5e09cdc (127.9.66.1). New cstate: current_term: 1 leader_uuid: "0b7b4fae549a422797b977bcf5e09cdc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c59f513dd13b45f4a554c64ff7a03c87" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 38723 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74c3871e1dde4f84bda59faeceb249fe" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 43023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b7b4fae549a422797b977bcf5e09cdc" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 37333 } health_report { overall_health: HEALTHY } } }
I20260925 07:32:53.097208 23971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:53.094874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50850 (local address 127.9.66.1:37333)
0925 07:32:53.094936 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 07:32:53.094938 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:32:53.094947 (+ 9us) server_negotiation.cc:407] Connection header received
0925 07:32:53.094992 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:32:53.094993 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:32:53.095010 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:32:53.095070 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 07:32:53.095184 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:32:53.095703 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:32:53.096415 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:32:53.096641 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:32:53.096709 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:32:53.096825 (+ 116us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 07:32:53.096929 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 07:32:53.097000 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 07:32:53.097039 (+ 39us) server_negotiation.cc:299] Negotiation successful
0925 07:32:53.097064 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 07:32:53.098894 24187 raft_consensus.cc:1261] T ab88041be7cd4c69aa6c46a5890553c9 P 74c3871e1dde4f84bda59faeceb249fe [term 1 FOLLOWER]: Refusing update from remote peer 0b7b4fae549a422797b977bcf5e09cdc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:32:53.098929 24055 raft_consensus.cc:1261] T ab88041be7cd4c69aa6c46a5890553c9 P c59f513dd13b45f4a554c64ff7a03c87 [term 1 FOLLOWER]: Refusing update from remote peer 0b7b4fae549a422797b977bcf5e09cdc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:32:53.099243 23969 consensus_queue.cc:1035] T ab88041be7cd4c69aa6c46a5890553c9 P 0b7b4fae549a422797b977bcf5e09cdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "74c3871e1dde4f84bda59faeceb249fe" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 43023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:32:53.099424 24250 consensus_queue.cc:1035] T ab88041be7cd4c69aa6c46a5890553c9 P 0b7b4fae549a422797b977bcf5e09cdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c59f513dd13b45f4a554c64ff7a03c87" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 38723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:32:53.102067 24253 mvcc.cc:205] Tried to move back new op lower bound from 7333157163410288640 to 7333157163265257472. Current Snapshot: MvccSnapshot[applied={T|T < 7333157163410288640}]
I20260925 07:32:53.102993 24254 mvcc.cc:205] Tried to move back new op lower bound from 7333157163410288640 to 7333157163265257472. Current Snapshot: MvccSnapshot[applied={T|T < 7333157163410288640 or (T in {7333157163410288640})}]
I20260925 07:32:53.104313 23774 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:37788:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 07:32:53.104408 23774 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:37788:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 07:32:53.105163 23774 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 265a353e5527411ca0c035891c95121e: Sending DeleteTablet for 3 replicas of tablet ab88041be7cd4c69aa6c46a5890553c9
I20260925 07:32:53.105466 24167 tablet_service.cc:1514] Processing DeleteTablet for tablet ab88041be7cd4c69aa6c46a5890553c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:32:53 UTC) from {username='kudu', principal='kudu/127.9.66.62@KRBTEST.COM'} at 127.0.0.1:53664
I20260925 07:32:53.105499 23897 tablet_service.cc:1514] Processing DeleteTablet for tablet ab88041be7cd4c69aa6c46a5890553c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:32:53 UTC) from {username='kudu', principal='kudu/127.9.66.62@KRBTEST.COM'} at 127.0.0.1:50844
I20260925 07:32:53.105487 24035 tablet_service.cc:1514] Processing DeleteTablet for tablet ab88041be7cd4c69aa6c46a5890553c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:32:53 UTC) from {username='kudu', principal='kudu/127.9.66.62@KRBTEST.COM'} at 127.0.0.1:37250
I20260925 07:32:53.105598 23972 tablet_replica.cc:330] T ab88041be7cd4c69aa6c46a5890553c9 P 0b7b4fae549a422797b977bcf5e09cdc: stopping tablet replica
I20260925 07:32:53.105649 23972 raft_consensus.cc:2229] T ab88041be7cd4c69aa6c46a5890553c9 P 0b7b4fae549a422797b977bcf5e09cdc [term 1 LEADER]: Raft consensus shutting down.
I20260925 07:32:53.105706 24264 tablet_replica.cc:330] T ab88041be7cd4c69aa6c46a5890553c9 P 74c3871e1dde4f84bda59faeceb249fe: stopping tablet replica
I20260925 07:32:53.105724 23972 raft_consensus.cc:2258] T ab88041be7cd4c69aa6c46a5890553c9 P 0b7b4fae549a422797b977bcf5e09cdc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:32:53.105770 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 23814
I20260925 07:32:53.105788 24264 raft_consensus.cc:2229] T ab88041be7cd4c69aa6c46a5890553c9 P 74c3871e1dde4f84bda59faeceb249fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 07:32:53.106362 24264 raft_consensus.cc:2258] T ab88041be7cd4c69aa6c46a5890553c9 P 74c3871e1dde4f84bda59faeceb249fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:32:53.106850 24264 ts_tablet_manager.cc:1905] T ab88041be7cd4c69aa6c46a5890553c9 P 74c3871e1dde4f84bda59faeceb249fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 07:32:53.108834 24264 ts_tablet_manager.cc:1918] T ab88041be7cd4c69aa6c46a5890553c9 P 74c3871e1dde4f84bda59faeceb249fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 07:32:53.108911 24264 log.cc:1196] T ab88041be7cd4c69aa6c46a5890553c9 P 74c3871e1dde4f84bda59faeceb249fe: Deleting WAL directory at /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-2/wal/wals/ab88041be7cd4c69aa6c46a5890553c9
I20260925 07:32:53.109177 24264 ts_tablet_manager.cc:1939] T ab88041be7cd4c69aa6c46a5890553c9 P 74c3871e1dde4f84bda59faeceb249fe: Deleting consensus metadata
I20260925 07:32:53.109616 23760 catalog_manager.cc:4841] TS 74c3871e1dde4f84bda59faeceb249fe (127.9.66.3:43023): tablet ab88041be7cd4c69aa6c46a5890553c9 (table test-table [id=6388dc24096d4539bd3fd161abc852b5]) successfully deleted
W20260925 07:32:53.111192 23761 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.66.1:37333 (error 108)
I20260925 07:32:53.111378 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 23968
I20260925 07:32:53.111402 24263 tablet_replica.cc:330] T ab88041be7cd4c69aa6c46a5890553c9 P c59f513dd13b45f4a554c64ff7a03c87: stopping tablet replica
I20260925 07:32:53.113198 24239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:53.113036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.1:37333 (local address 127.0.0.1:50860)
0925 07:32:53.113117 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0925 07:32:53.113159 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.1:37333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260925 07:32:53.113329 23761 catalog_manager.cc:4584] TS 0b7b4fae549a422797b977bcf5e09cdc (127.9.66.1:37333): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ab88041be7cd4c69aa6c46a5890553c9: Network error: Client connection negotiation failed: client connection to 127.9.66.1:37333: connect: Connection refused (error 111)
I20260925 07:32:53.116286 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 24104
I20260925 07:32:53.116335 24239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:53.116183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.2:38723 (local address 127.0.0.1:37264)
0925 07:32:53.116260 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0925 07:32:53.116311 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.2:38723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260925 07:32:53.120823 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 23742
I20260925 07:32:53.125953 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.66.62:46577
--webserver_interface=127.9.66.62
--webserver_port=35781
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab
--principal=kudu/127.9.66.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:35859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.66.62:46577
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:32:53.204460 24266 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:32:53.204631 24266 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:32:53.204651 24266 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:32:53.205761 24266 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:32:53.205798 24266 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:32:53.205812 24266 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:32:53.205824 24266 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:32:53.205863 24266 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:32:53.207274 24266 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:35859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.66.62:46577
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab
--principal=kudu/127.9.66.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.66.62:46577
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.66.62
--webserver_port=35781
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24266
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:32:53.207500 24266 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:32:53.207726 24266 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:32:53.212574 24272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:32:53.212615 24266 server_base.cc:1034] running on GCE node
W20260925 07:32:53.212580 24271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:32:53.212579 24274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:32:53.212973 24266 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:32:53.213243 24266 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:32:53.214375 24266 hybrid_clock.cc:648] HybridClock initialized: now 1790321573214358 us; error 33 us; skew 500 ppm
Sep 25 07:32:53 dist-test-slave-57fn krb5kdc[23198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321573, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:32:53.216591 24266 init.cc:377] Logged in from keytab as kudu/127.9.66.62@KRBTEST.COM (short username kudu)
I20260925 07:32:53.217015 24266 webserver.cc:460] Webserver started at http://127.9.66.62:35781/ using document root <none> and password file <none>
I20260925 07:32:53.217221 24266 fs_manager.cc:359] Metadata directory not provided
I20260925 07:32:53.217262 24266 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:32:53.218500 24266 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:32:53.219125 24281 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:32:53.219292 24266 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 07:32:53.219400 24266 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/master-0/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/master-0/wal
uuid: "265a353e5527411ca0c035891c95121e"
format_stamp: "Formatted at 2026-09-25 07:32:51 on dist-test-slave-57fn"
server_key: "4f72d3af8ddb6c5ae0dd1db167acd81a"
server_key_iv: "eacc4d534c2689883e186a27aea4d983"
server_key_version: "encryptionkey@0"
I20260925 07:32:53.219732 24266 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:32:53.227298 24266 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:32:53.228960 24266 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:32:53.229104 24266 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:32:53.232957 24266 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.62:46577
I20260925 07:32:53.232986 24333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.62:46577 every 8 connection(s)
I20260925 07:32:53.233331 24266 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
I20260925 07:32:53.234783 24334 sys_catalog.cc:263] Verifying existing consensus state
I20260925 07:32:53.235347 24334 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 265a353e5527411ca0c035891c95121e: Bootstrap starting.
I20260925 07:32:53.240370 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 24266
I20260925 07:32:53.240691 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.66.1:37333
--local_ip_for_outbound_sockets=127.9.66.1
--tserver_master_addrs=127.9.66.62:46577
--webserver_port=46549
--webserver_interface=127.9.66.1
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.1.keytab
--principal=kudu/127.9.66.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:35859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 07:32:53.243800 24334 log.cc:824] T 00000000000000000000000000000000 P 265a353e5527411ca0c035891c95121e: Log is configured to *not* fsync() on all Append() calls
I20260925 07:32:53.245926 24334 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 265a353e5527411ca0c035891c95121e: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 07:32:53.246206 24334 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 265a353e5527411ca0c035891c95121e: Bootstrap complete.
I20260925 07:32:53.248330 24334 raft_consensus.cc:348] T 00000000000000000000000000000000 P 265a353e5527411ca0c035891c95121e [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "265a353e5527411ca0c035891c95121e" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 46577 } }
I20260925 07:32:53.248546 24334 raft_consensus.cc:729] T 00000000000000000000000000000000 P 265a353e5527411ca0c035891c95121e [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 265a353e5527411ca0c035891c95121e, State: Initialized, Role: FOLLOWER
I20260925 07:32:53.248709 24334 consensus_queue.cc:260] T 00000000000000000000000000000000 P 265a353e5527411ca0c035891c95121e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "265a353e5527411ca0c035891c95121e" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 46577 } }
I20260925 07:32:53.248766 24334 raft_consensus.cc:388] T 00000000000000000000000000000000 P 265a353e5527411ca0c035891c95121e [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 07:32:53.248824 24334 raft_consensus.cc:482] T 00000000000000000000000000000000 P 265a353e5527411ca0c035891c95121e [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 07:32:53.248869 24334 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 265a353e5527411ca0c035891c95121e [term 2 FOLLOWER]: Advancing to term 3
I20260925 07:32:53.249907 24334 raft_consensus.cc:504] T 00000000000000000000000000000000 P 265a353e5527411ca0c035891c95121e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "265a353e5527411ca0c035891c95121e" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 46577 } }
I20260925 07:32:53.249989 24334 leader_election.cc:302] T 00000000000000000000000000000000 P 265a353e5527411ca0c035891c95121e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 265a353e5527411ca0c035891c95121e; no voters:
I20260925 07:32:53.250216 24334 leader_election.cc:288] T 00000000000000000000000000000000 P 265a353e5527411ca0c035891c95121e [CANDIDATE]: Term 3 election: Requested vote from peers
I20260925 07:32:53.250248 24338 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 265a353e5527411ca0c035891c95121e [term 3 FOLLOWER]: Leader election won for term 3
I20260925 07:32:53.250347 24338 raft_consensus.cc:686] T 00000000000000000000000000000000 P 265a353e5527411ca0c035891c95121e [term 3 LEADER]: Becoming Leader. State: Replica: 265a353e5527411ca0c035891c95121e, State: Running, Role: LEADER
I20260925 07:32:53.250432 24338 consensus_queue.cc:237] T 00000000000000000000000000000000 P 265a353e5527411ca0c035891c95121e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "265a353e5527411ca0c035891c95121e" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 46577 } }
I20260925 07:32:53.250463 24334 sys_catalog.cc:564] T 00000000000000000000000000000000 P 265a353e5527411ca0c035891c95121e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:32:53.250661 24338 sys_catalog.cc:455] T 00000000000000000000000000000000 P 265a353e5527411ca0c035891c95121e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "265a353e5527411ca0c035891c95121e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "265a353e5527411ca0c035891c95121e" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 46577 } } }
I20260925 07:32:53.250720 24338 sys_catalog.cc:458] T 00000000000000000000000000000000 P 265a353e5527411ca0c035891c95121e [sys.catalog]: This master's current role is: LEADER
I20260925 07:32:53.250818 24338 sys_catalog.cc:455] T 00000000000000000000000000000000 P 265a353e5527411ca0c035891c95121e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 265a353e5527411ca0c035891c95121e. Latest consensus state: current_term: 3 leader_uuid: "265a353e5527411ca0c035891c95121e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "265a353e5527411ca0c035891c95121e" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 46577 } } }
I20260925 07:32:53.250877 24338 sys_catalog.cc:458] T 00000000000000000000000000000000 P 265a353e5527411ca0c035891c95121e [sys.catalog]: This master's current role is: LEADER
I20260925 07:32:53.250981 24342 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 07:32:53.251731 24342 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 07:32:53.251827 24342 catalog_manager.cc:1259] Loaded cluster ID: 0801089512b5455085114a5cfb31516e
I20260925 07:32:53.251856 24342 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260925 07:32:53.252072 24342 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fcf5703d9ad google::LogMessage::Fail() at ??:0
@ 0x7fcf5704088a google::LogMessage::SendToLog() at ??:0
@ 0x7fcf5703d4de google::LogMessage::Flush() at ??:0
@ 0x7fcf5703ea19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fcf5ef5e0f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fcf5ef678b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fcf58f0b0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fcf58f00954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fcf59d646db start_thread at ??:0
@ 0x7fcf55b3b61f clone at ??:0
W20260925 07:32:53.332410 24336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:32:53.332629 24336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:32:53.332662 24336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:32:53.333781 24336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:32:53.333853 24336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:32:53.333899 24336 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.66.1
I20260925 07:32:53.335456 24336 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:35859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.1.keytab
--principal=kudu/127.9.66.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.66.1:37333
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.66.1
--webserver_port=46549
--tserver_master_addrs=127.9.66.62:46577
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24336
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.66.1
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:32:53.335706 24336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:32:53.335918 24336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:32:53.340739 24370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:32:53.340902 24371 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:32:53.340770 24373 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:32:53.341245 24336 server_base.cc:1034] running on GCE node
I20260925 07:32:53.341464 24336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:32:53.341751 24336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:32:53.342936 24336 hybrid_clock.cc:648] HybridClock initialized: now 1790321573342918 us; error 43 us; skew 500 ppm
Sep 25 07:32:53 dist-test-slave-57fn krb5kdc[23198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321573, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:32:53.346238 24336 init.cc:377] Logged in from keytab as kudu/127.9.66.1@KRBTEST.COM (short username kudu)
I20260925 07:32:53.346688 24336 webserver.cc:460] Webserver started at http://127.9.66.1:46549/ using document root <none> and password file <none>
I20260925 07:32:53.346915 24336 fs_manager.cc:359] Metadata directory not provided
I20260925 07:32:53.346982 24336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:32:53.348353 24336 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:32:53.349123 24380 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:32:53.349303 24336 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:32:53.349399 24336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-0/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-0/wal
uuid: "0b7b4fae549a422797b977bcf5e09cdc"
format_stamp: "Formatted at 2026-09-25 07:32:51 on dist-test-slave-57fn"
server_key: "85dcf2e91ba7b87705987c903cabf23b"
server_key_iv: "52b33f563b20366aa695e29ac5be35ff"
server_key_version: "encryptionkey@0"
I20260925 07:32:53.349800 24336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:32:53.374716 24336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:32:53.377069 24336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:32:53.377239 24336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:32:53.377480 24336 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:32:53.377987 24387 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 07:32:53.380187 24336 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 07:32:53.380272 24336 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260925 07:32:53.380321 24336 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 07:32:53.380932 24336 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 07:32:53.380986 24336 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260925 07:32:53.381080 24387 tablet_bootstrap.cc:492] T ab88041be7cd4c69aa6c46a5890553c9 P 0b7b4fae549a422797b977bcf5e09cdc: Bootstrap starting.
I20260925 07:32:53.388134 24336 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.1:37333
I20260925 07:32:53.388228 24494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.1:37333 every 8 connection(s)
I20260925 07:32:53.388540 24336 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-0/data/info.pb
I20260925 07:32:53.392800 24387 log.cc:824] T ab88041be7cd4c69aa6c46a5890553c9 P 0b7b4fae549a422797b977bcf5e09cdc: Log is configured to *not* fsync() on all Append() calls
I20260925 07:32:53.395437 24387 tablet_bootstrap.cc:492] T ab88041be7cd4c69aa6c46a5890553c9 P 0b7b4fae549a422797b977bcf5e09cdc: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 07:32:53.395751 24387 tablet_bootstrap.cc:492] T ab88041be7cd4c69aa6c46a5890553c9 P 0b7b4fae549a422797b977bcf5e09cdc: Bootstrap complete.
I20260925 07:32:53.396055 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 24336
I20260925 07:32:53.396301 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.66.2:38723
--local_ip_for_outbound_sockets=127.9.66.2
--tserver_master_addrs=127.9.66.62:46577
--webserver_port=35775
--webserver_interface=127.9.66.2
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.2.keytab
--principal=kudu/127.9.66.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:35859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 07:32:53.397156 24387 ts_tablet_manager.cc:1397] T ab88041be7cd4c69aa6c46a5890553c9 P 0b7b4fae549a422797b977bcf5e09cdc: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.009s
I20260925 07:32:53.398698 24387 raft_consensus.cc:348] T ab88041be7cd4c69aa6c46a5890553c9 P 0b7b4fae549a422797b977bcf5e09cdc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c59f513dd13b45f4a554c64ff7a03c87" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 38723 } } peers { permanent_uuid: "74c3871e1dde4f84bda59faeceb249fe" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 43023 } } peers { permanent_uuid: "0b7b4fae549a422797b977bcf5e09cdc" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 37333 } }
I20260925 07:32:53.399219 24387 raft_consensus.cc:729] T ab88041be7cd4c69aa6c46a5890553c9 P 0b7b4fae549a422797b977bcf5e09cdc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b7b4fae549a422797b977bcf5e09cdc, State: Initialized, Role: FOLLOWER
I20260925 07:32:53.399379 24387 consensus_queue.cc:260] T ab88041be7cd4c69aa6c46a5890553c9 P 0b7b4fae549a422797b977bcf5e09cdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c59f513dd13b45f4a554c64ff7a03c87" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 38723 } } peers { permanent_uuid: "74c3871e1dde4f84bda59faeceb249fe" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 43023 } } peers { permanent_uuid: "0b7b4fae549a422797b977bcf5e09cdc" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 37333 } }
I20260925 07:32:53.399647 24387 ts_tablet_manager.cc:1428] T ab88041be7cd4c69aa6c46a5890553c9 P 0b7b4fae549a422797b977bcf5e09cdc: Time spent starting tablet: real 0.002s user 0.003s sys 0.003s
W20260925 07:32:53.478044 24499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:32:53.478343 24499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:32:53.478384 24499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:32:53.480938 24499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:32:53.480996 24499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:32:53.481046 24499 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.66.2
W20260925 07:32:53.481742 24497 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:32:53.389203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:46577 (local address 127.9.66.1:50405)
0925 07:32:53.389373 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 07:32:53.389412 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 07:32:53.389885 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:32:53.481455 (+ 91570us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:46577: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":93,"threads_started":1}
W20260925 07:32:53.482010 24495 heartbeater.cc:644] Failed to heartbeat to 127.9.66.62:46577 (0 consecutive failures): Network error: Failed to ping master at 127.9.66.62:46577: Client connection negotiation failed: client connection to 127.9.66.62:46577: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 07:32:53.482352 24497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:53.482235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:46577 (local address 127.9.66.1:35981)
0925 07:32:53.482289 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0925 07:32:53.482325 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:46577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260925 07:32:53.482703 24497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:53.482618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:46577 (local address 127.9.66.1:35331)
0925 07:32:53.482665 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0925 07:32:53.482688 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:46577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260925 07:32:53.482829 24495 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 07:32:53.483112 24499 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:35859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.2.keytab
--principal=kudu/127.9.66.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.66.2:38723
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.66.2
--webserver_port=35775
--tserver_master_addrs=127.9.66.62:46577
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24499
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.66.2
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:32:53.483311 24499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:32:53.483500 24499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:32:53.488312 24505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:32:53.488305 24506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:32:53.488446 24499 server_base.cc:1034] running on GCE node
W20260925 07:32:53.488305 24508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:32:53.488692 24499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:32:53.488951 24499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:32:53.490092 24499 hybrid_clock.cc:648] HybridClock initialized: now 1790321573490079 us; error 28 us; skew 500 ppm
Sep 25 07:32:53 dist-test-slave-57fn krb5kdc[23198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321573, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:32:53.492501 24499 init.cc:377] Logged in from keytab as kudu/127.9.66.2@KRBTEST.COM (short username kudu)
I20260925 07:32:53.492938 24499 webserver.cc:460] Webserver started at http://127.9.66.2:35775/ using document root <none> and password file <none>
I20260925 07:32:53.493131 24499 fs_manager.cc:359] Metadata directory not provided
I20260925 07:32:53.493171 24499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:32:53.494488 24499 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:32:53.495097 24515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:32:53.495289 24499 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 07:32:53.495368 24499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-1/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-1/wal
uuid: "c59f513dd13b45f4a554c64ff7a03c87"
format_stamp: "Formatted at 2026-09-25 07:32:51 on dist-test-slave-57fn"
server_key: "f2804f27d213bc1967c47cb055e357d5"
server_key_iv: "0cc2b036bc76740d6e03c407966f7a62"
server_key_version: "encryptionkey@0"
I20260925 07:32:53.495692 24499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:32:53.514312 24499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:32:53.516188 24499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:32:53.516296 24499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:32:53.516467 24499 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:32:53.516821 24522 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 07:32:53.518280 24499 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 07:32:53.518322 24499 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260925 07:32:53.518366 24499 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 07:32:53.518932 24499 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 07:32:53.518963 24499 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260925 07:32:53.519049 24522 tablet_bootstrap.cc:492] T ab88041be7cd4c69aa6c46a5890553c9 P c59f513dd13b45f4a554c64ff7a03c87: Bootstrap starting.
I20260925 07:32:53.525086 24499 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.2:38723
I20260925 07:32:53.525193 24629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.2:38723 every 8 connection(s)
I20260925 07:32:53.525467 24499 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-1/data/info.pb
I20260925 07:32:53.526708 24632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:53.526220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:46577 (local address 127.9.66.2:34995)
0925 07:32:53.526415 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0925 07:32:53.526508 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:46577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":73,"threads_started":1}
W20260925 07:32:53.526943 24630 heartbeater.cc:644] Failed to heartbeat to 127.9.66.62:46577 (0 consecutive failures): Network error: Failed to ping master at 127.9.66.62:46577: Client connection negotiation failed: client connection to 127.9.66.62:46577: connect: Connection refused (error 111)
I20260925 07:32:53.527257 24632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:53.527149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:46577 (local address 127.9.66.2:41547)
0925 07:32:53.527199 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0925 07:32:53.527231 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:46577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260925 07:32:53.527604 24632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:53.527476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:46577 (local address 127.9.66.2:38873)
0925 07:32:53.527537 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0925 07:32:53.527572 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:46577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260925 07:32:53.527770 24630 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 07:32:53.529834 24522 log.cc:824] T ab88041be7cd4c69aa6c46a5890553c9 P c59f513dd13b45f4a554c64ff7a03c87: Log is configured to *not* fsync() on all Append() calls
I20260925 07:32:53.531814 24522 tablet_bootstrap.cc:492] T ab88041be7cd4c69aa6c46a5890553c9 P c59f513dd13b45f4a554c64ff7a03c87: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 07:32:53.532083 24522 tablet_bootstrap.cc:492] T ab88041be7cd4c69aa6c46a5890553c9 P c59f513dd13b45f4a554c64ff7a03c87: Bootstrap complete.
I20260925 07:32:53.532404 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 24499
I20260925 07:32:53.532634 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.66.3:43023
--local_ip_for_outbound_sockets=127.9.66.3
--tserver_master_addrs=127.9.66.62:46577
--webserver_port=45283
--webserver_interface=127.9.66.3
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.3.keytab
--principal=kudu/127.9.66.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:35859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 07:32:53.533370 24522 ts_tablet_manager.cc:1397] T ab88041be7cd4c69aa6c46a5890553c9 P c59f513dd13b45f4a554c64ff7a03c87: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.008s
I20260925 07:32:53.534906 24522 raft_consensus.cc:348] T ab88041be7cd4c69aa6c46a5890553c9 P c59f513dd13b45f4a554c64ff7a03c87 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c59f513dd13b45f4a554c64ff7a03c87" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 38723 } } peers { permanent_uuid: "74c3871e1dde4f84bda59faeceb249fe" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 43023 } } peers { permanent_uuid: "0b7b4fae549a422797b977bcf5e09cdc" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 37333 } }
I20260925 07:32:53.535349 24522 raft_consensus.cc:729] T ab88041be7cd4c69aa6c46a5890553c9 P c59f513dd13b45f4a554c64ff7a03c87 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c59f513dd13b45f4a554c64ff7a03c87, State: Initialized, Role: FOLLOWER
I20260925 07:32:53.535490 24522 consensus_queue.cc:260] T ab88041be7cd4c69aa6c46a5890553c9 P c59f513dd13b45f4a554c64ff7a03c87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c59f513dd13b45f4a554c64ff7a03c87" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 38723 } } peers { permanent_uuid: "74c3871e1dde4f84bda59faeceb249fe" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 43023 } } peers { permanent_uuid: "0b7b4fae549a422797b977bcf5e09cdc" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 37333 } }
I20260925 07:32:53.535759 24522 ts_tablet_manager.cc:1428] T ab88041be7cd4c69aa6c46a5890553c9 P c59f513dd13b45f4a554c64ff7a03c87: Time spent starting tablet: real 0.002s user 0.004s sys 0.001s
I20260925 07:32:53.536085 24632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:53.535958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:46577 (local address 127.9.66.2:60499)
0925 07:32:53.536019 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0925 07:32:53.536054 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:46577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260925 07:32:53.536427 24632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:53.536306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:46577 (local address 127.9.66.2:55581)
0925 07:32:53.536371 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0925 07:32:53.536400 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:46577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260925 07:32:53.612886 24634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:32:53.613085 24634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:32:53.613108 24634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:32:53.614326 24634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:32:53.614367 24634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:32:53.614406 24634 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.66.3
I20260925 07:32:53.615770 24634 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:35859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.3.keytab
--principal=kudu/127.9.66.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.66.3:43023
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.66.3
--webserver_port=45283
--tserver_master_addrs=127.9.66.62:46577
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24634
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.66.3
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:32:53.615964 24634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:32:53.616185 24634 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:32:53.620925 24641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:32:53.620993 24643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:32:53.620940 24640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:32:53.621296 24634 server_base.cc:1034] running on GCE node
I20260925 07:32:53.621452 24634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:32:53.621891 24634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:32:53.623095 24634 hybrid_clock.cc:648] HybridClock initialized: now 1790321573623075 us; error 31 us; skew 500 ppm
Sep 25 07:32:53 dist-test-slave-57fn krb5kdc[23198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321573, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:32:53.625346 24634 init.cc:377] Logged in from keytab as kudu/127.9.66.3@KRBTEST.COM (short username kudu)
I20260925 07:32:53.625739 24634 webserver.cc:460] Webserver started at http://127.9.66.3:45283/ using document root <none> and password file <none>
I20260925 07:32:53.625898 24634 fs_manager.cc:359] Metadata directory not provided
I20260925 07:32:53.625934 24634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:32:53.627041 24634 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:32:53.627544 24650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:32:53.627722 24634 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 07:32:53.627804 24634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-2/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-2/wal
uuid: "74c3871e1dde4f84bda59faeceb249fe"
format_stamp: "Formatted at 2026-09-25 07:32:51 on dist-test-slave-57fn"
server_key: "f3df166509c3aac269bd51f1d78d232d"
server_key_iv: "4cccb95ed696f0c469ea187ac84a5f5a"
server_key_version: "encryptionkey@0"
I20260925 07:32:53.628116 24634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:32:53.651188 24634 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:32:53.653270 24634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:32:53.653417 24634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:32:53.653630 24634 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:32:53.653914 24634 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:32:53.653951 24634 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:32:53.653998 24634 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:32:53.654023 24634 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:32:53.659268 24634 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.3:43023
I20260925 07:32:53.659334 24763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.3:43023 every 8 connection(s)
I20260925 07:32:53.659693 24634 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790321392278483-9480-0/minicluster-data/ts-2/data/info.pb
I20260925 07:32:53.660691 24766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:53.660372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:46577 (local address 127.9.66.3:45435)
0925 07:32:53.660501 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0925 07:32:53.660563 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:46577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":62,"threads_started":1}
W20260925 07:32:53.661098 24764 heartbeater.cc:644] Failed to heartbeat to 127.9.66.62:46577 (0 consecutive failures): Network error: Failed to ping master at 127.9.66.62:46577: Client connection negotiation failed: client connection to 127.9.66.62:46577: connect: Connection refused (error 111)
I20260925 07:32:53.661345 24766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:53.661259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:46577 (local address 127.9.66.3:44427)
0925 07:32:53.661303 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0925 07:32:53.661328 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:46577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260925 07:32:53.661651 24766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:53.661599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:46577 (local address 127.9.66.3:41163)
0925 07:32:53.661620 (+ 21us) negotiation.cc:107] Waiting for socket to connect
0925 07:32:53.661639 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:46577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12}
W20260925 07:32:53.661761 24764 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 07:32:53.668145 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 24634
I20260925 07:32:54.483363 24768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:54.483146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:46577 (local address 127.9.66.1:59419)
0925 07:32:54.483273 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0925 07:32:54.483304 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:46577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":59,"threads_started":1}
I20260925 07:32:54.537103 24769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:54.536890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:46577 (local address 127.9.66.2:53409)
0925 07:32:54.537019 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0925 07:32:54.537050 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:46577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":58,"threads_started":1}
I20260925 07:32:54.662248 24770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:54.662062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:46577 (local address 127.9.66.3:59811)
0925 07:32:54.662168 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0925 07:32:54.662202 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:46577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":52,"threads_started":1}
I20260925 07:32:54.671530 24771 raft_consensus.cc:482] T ab88041be7cd4c69aa6c46a5890553c9 P 0b7b4fae549a422797b977bcf5e09cdc [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 07:32:54.671694 24771 raft_consensus.cc:504] T ab88041be7cd4c69aa6c46a5890553c9 P 0b7b4fae549a422797b977bcf5e09cdc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c59f513dd13b45f4a554c64ff7a03c87" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 38723 } } peers { permanent_uuid: "74c3871e1dde4f84bda59faeceb249fe" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 43023 } } peers { permanent_uuid: "0b7b4fae549a422797b977bcf5e09cdc" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 37333 } }
I20260925 07:32:54.671988 24771 leader_election.cc:288] T ab88041be7cd4c69aa6c46a5890553c9 P 0b7b4fae549a422797b977bcf5e09cdc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c59f513dd13b45f4a554c64ff7a03c87 (127.9.66.2:38723), 74c3871e1dde4f84bda59faeceb249fe (127.9.66.3:43023)
Sep 25 07:32:54 dist-test-slave-57fn krb5kdc[23198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321573, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.1@KRBTEST.COM for kudu/127.9.66.3@KRBTEST.COM
Sep 25 07:32:54 dist-test-slave-57fn krb5kdc[23198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321573, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.1@KRBTEST.COM for kudu/127.9.66.2@KRBTEST.COM
I20260925 07:32:54.681433 24768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:54.672127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.3:43023 (local address 127.9.66.1:60191)
0925 07:32:54.672185 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0925 07:32:54.672195 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 07:32:54.672246 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:32:54.673112 (+ 866us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:32:54.673113 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:32:54.673332 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:32:54.673428 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:32:54.673434 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:32:54.674512 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:32:54.674513 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:32:54.675253 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:32:54.675255 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:32:54.675849 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:32:54.675850 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:32:54.675950 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:32:54.676408 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:32:54.676415 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:32:54.677744 (+ 1329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:32:54.679779 (+ 2035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:32:54.679786 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:32:54.679788 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:32:54.679994 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:32:54.680343 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:32:54.680344 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:32:54.680344 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:32:54.680465 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:32:54.681001 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:32:54.681006 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:32:54.681106 (+ 100us) client_negotiation.cc:769] Sending connection context
0925 07:32:54.681183 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 07:32:54.681324 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":3}
I20260925 07:32:54.681533 24773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:54.672221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.1:60191 (local address 127.9.66.3:43023)
0925 07:32:54.672356 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 07:32:54.672368 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 07:32:54.672404 (+ 36us) server_negotiation.cc:407] Connection header received
0925 07:32:54.672962 (+ 558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:32:54.672964 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:32:54.673004 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:32:54.673130 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:32:54.673582 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:32:54.674396 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:32:54.675355 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:32:54.675757 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:32:54.675872 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:32:54.677844 (+ 1972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:32:54.677848 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:32:54.677849 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:32:54.677861 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:32:54.679663 (+ 1802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:32:54.680108 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:32:54.680111 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:32:54.680111 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:32:54.680227 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:32:54.680552 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:32:54.680553 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:32:54.680553 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:32:54.680922 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:32:54.681013 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 07:32:54.681268 (+ 255us) server_negotiation.cc:299] Negotiation successful
0925 07:32:54.681404 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":67,"threads_started":1}
I20260925 07:32:54.681777 24772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:54.672127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.2:38723 (local address 127.9.66.1:44011)
0925 07:32:54.672423 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0925 07:32:54.672433 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 07:32:54.672496 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:32:54.672844 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:32:54.672847 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:32:54.673217 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:32:54.673429 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:32:54.673434 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:32:54.674543 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:32:54.674544 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:32:54.675254 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:32:54.675256 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:32:54.675938 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:32:54.675939 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:32:54.675990 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:32:54.676566 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:32:54.676577 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:32:54.678162 (+ 1585us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:32:54.680265 (+ 2103us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:32:54.680266 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:32:54.680266 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:32:54.680457 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:32:54.680763 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:32:54.680764 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:32:54.680764 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:32:54.680829 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:32:54.681278 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:32:54.681279 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:32:54.681318 (+ 39us) client_negotiation.cc:769] Sending connection context
0925 07:32:54.681379 (+ 61us) client_negotiation.cc:240] Negotiation successful
0925 07:32:54.681454 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"mutex_wait_us":72,"thread_start_us":62,"threads_started":1}
I20260925 07:32:54.681890 24774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:54.672268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.1:44011 (local address 127.9.66.2:38723)
0925 07:32:54.672394 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0925 07:32:54.672401 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 07:32:54.672498 (+ 97us) server_negotiation.cc:407] Connection header received
0925 07:32:54.672603 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:32:54.672605 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:32:54.672656 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:32:54.672792 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:32:54.673564 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:32:54.674414 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:32:54.675442 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:32:54.675836 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:32:54.675966 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:32:54.678273 (+ 2307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:32:54.678277 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:32:54.678277 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:32:54.678288 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:32:54.680116 (+ 1828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:32:54.680560 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:32:54.680562 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:32:54.680562 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:32:54.680678 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:32:54.680895 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:32:54.680896 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:32:54.680896 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:32:54.681162 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:32:54.681294 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0925 07:32:54.681446 (+ 152us) server_negotiation.cc:299] Negotiation successful
0925 07:32:54.681797 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":57,"threads_started":1}
W20260925 07:32:54.682226 24382 leader_election.cc:341] T ab88041be7cd4c69aa6c46a5890553c9 P 0b7b4fae549a422797b977bcf5e09cdc [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 74c3871e1dde4f84bda59faeceb249fe (127.9.66.3:43023): Not found: Tablet not found: ab88041be7cd4c69aa6c46a5890553c9
I20260925 07:32:54.682375 24584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab88041be7cd4c69aa6c46a5890553c9" candidate_uuid: "0b7b4fae549a422797b977bcf5e09cdc" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "c59f513dd13b45f4a554c64ff7a03c87" is_pre_election: true
I20260925 07:32:54.682534 24584 raft_consensus.cc:2454] T ab88041be7cd4c69aa6c46a5890553c9 P c59f513dd13b45f4a554c64ff7a03c87 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b7b4fae549a422797b977bcf5e09cdc in term 1.
I20260925 07:32:54.682744 24384 leader_election.cc:302] T ab88041be7cd4c69aa6c46a5890553c9 P 0b7b4fae549a422797b977bcf5e09cdc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b7b4fae549a422797b977bcf5e09cdc, c59f513dd13b45f4a554c64ff7a03c87; no voters: 74c3871e1dde4f84bda59faeceb249fe
I20260925 07:32:54.682876 24771 raft_consensus.cc:2781] T ab88041be7cd4c69aa6c46a5890553c9 P 0b7b4fae549a422797b977bcf5e09cdc [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 07:32:54.682960 24771 raft_consensus.cc:482] T ab88041be7cd4c69aa6c46a5890553c9 P 0b7b4fae549a422797b977bcf5e09cdc [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 07:32:54.682999 24771 raft_consensus.cc:3037] T ab88041be7cd4c69aa6c46a5890553c9 P 0b7b4fae549a422797b977bcf5e09cdc [term 1 FOLLOWER]: Advancing to term 2
I20260925 07:32:54.684124 24771 raft_consensus.cc:504] T ab88041be7cd4c69aa6c46a5890553c9 P 0b7b4fae549a422797b977bcf5e09cdc [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c59f513dd13b45f4a554c64ff7a03c87" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 38723 } } peers { permanent_uuid: "74c3871e1dde4f84bda59faeceb249fe" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 43023 } } peers { permanent_uuid: "0b7b4fae549a422797b977bcf5e09cdc" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 37333 } }
I20260925 07:32:54.684286 24771 leader_election.cc:288] T ab88041be7cd4c69aa6c46a5890553c9 P 0b7b4fae549a422797b977bcf5e09cdc [CANDIDATE]: Term 2 election: Requested vote from peers c59f513dd13b45f4a554c64ff7a03c87 (127.9.66.2:38723), 74c3871e1dde4f84bda59faeceb249fe (127.9.66.3:43023)
I20260925 07:32:54.684442 24584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab88041be7cd4c69aa6c46a5890553c9" candidate_uuid: "0b7b4fae549a422797b977bcf5e09cdc" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "c59f513dd13b45f4a554c64ff7a03c87"
W20260925 07:32:54.684541 24382 leader_election.cc:341] T ab88041be7cd4c69aa6c46a5890553c9 P 0b7b4fae549a422797b977bcf5e09cdc [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 74c3871e1dde4f84bda59faeceb249fe (127.9.66.3:43023): Not found: Tablet not found: ab88041be7cd4c69aa6c46a5890553c9
I20260925 07:32:54.684545 24584 raft_consensus.cc:3037] T ab88041be7cd4c69aa6c46a5890553c9 P c59f513dd13b45f4a554c64ff7a03c87 [term 1 FOLLOWER]: Advancing to term 2
I20260925 07:32:54.685648 24584 raft_consensus.cc:2454] T ab88041be7cd4c69aa6c46a5890553c9 P c59f513dd13b45f4a554c64ff7a03c87 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b7b4fae549a422797b977bcf5e09cdc in term 2.
I20260925 07:32:54.685941 24384 leader_election.cc:302] T ab88041be7cd4c69aa6c46a5890553c9 P 0b7b4fae549a422797b977bcf5e09cdc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b7b4fae549a422797b977bcf5e09cdc, c59f513dd13b45f4a554c64ff7a03c87; no voters: 74c3871e1dde4f84bda59faeceb249fe
I20260925 07:32:54.686067 24771 raft_consensus.cc:2781] T ab88041be7cd4c69aa6c46a5890553c9 P 0b7b4fae549a422797b977bcf5e09cdc [term 2 FOLLOWER]: Leader election won for term 2
I20260925 07:32:54.686199 24771 raft_consensus.cc:686] T ab88041be7cd4c69aa6c46a5890553c9 P 0b7b4fae549a422797b977bcf5e09cdc [term 2 LEADER]: Becoming Leader. State: Replica: 0b7b4fae549a422797b977bcf5e09cdc, State: Running, Role: LEADER
I20260925 07:32:54.686302 24771 consensus_queue.cc:237] T ab88041be7cd4c69aa6c46a5890553c9 P 0b7b4fae549a422797b977bcf5e09cdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c59f513dd13b45f4a554c64ff7a03c87" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 38723 } } peers { permanent_uuid: "74c3871e1dde4f84bda59faeceb249fe" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 43023 } } peers { permanent_uuid: "0b7b4fae549a422797b977bcf5e09cdc" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 37333 } }
I20260925 07:32:54.686512 24768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:54.686397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:46577 (local address 127.9.66.1:57675)
0925 07:32:54.686445 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0925 07:32:54.686476 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:46577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260925 07:32:55.121606 24584 raft_consensus.cc:1261] T ab88041be7cd4c69aa6c46a5890553c9 P c59f513dd13b45f4a554c64ff7a03c87 [term 2 FOLLOWER]: Refusing update from remote peer 0b7b4fae549a422797b977bcf5e09cdc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260925 07:32:55.122020 24771 consensus_queue.cc:1035] T ab88041be7cd4c69aa6c46a5890553c9 P 0b7b4fae549a422797b977bcf5e09cdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c59f513dd13b45f4a554c64ff7a03c87" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 38723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260925 07:32:55.122849 24779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:55.122684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:46577 (local address 127.9.66.2:36961)
0925 07:32:55.122784 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0925 07:32:55.122814 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:46577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":46,"threads_started":1}
I20260925 07:32:55.123735 24779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:55.123637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:46577 (local address 127.9.66.2:39603)
0925 07:32:55.123676 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0925 07:32:55.123706 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:46577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260925 07:32:55.123762 24768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:55.123650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:46577 (local address 127.9.66.1:37331)
0925 07:32:55.123711 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0925 07:32:55.123731 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:46577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260925 07:32:55.123907 24382 consensus_peers.cc:487] T ab88041be7cd4c69aa6c46a5890553c9 P 0b7b4fae549a422797b977bcf5e09cdc -> Peer 74c3871e1dde4f84bda59faeceb249fe (127.9.66.3:43023): Couldn't send request to peer 74c3871e1dde4f84bda59faeceb249fe. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ab88041be7cd4c69aa6c46a5890553c9. This is attempt 1: this message will repeat every 5th retry.
I20260925 07:32:55.662946 24783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:55.662756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:46577 (local address 127.9.66.3:48457)
0925 07:32:55.662876 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0925 07:32:55.662910 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:46577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":56,"threads_started":1}
I20260925 07:32:55.708595 24784 ts_tablet_manager.cc:927] T ab88041be7cd4c69aa6c46a5890553c9 P 74c3871e1dde4f84bda59faeceb249fe: Initiating tablet copy from peer 0b7b4fae549a422797b977bcf5e09cdc (127.9.66.1:37333)
I20260925 07:32:55.708832 24784 tablet_copy_client.cc:316] T ab88041be7cd4c69aa6c46a5890553c9 P 74c3871e1dde4f84bda59faeceb249fe: tablet copy: Beginning tablet copy session from remote peer at address 127.9.66.1:37333
Sep 25 07:32:55 dist-test-slave-57fn krb5kdc[23198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321573, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.3@KRBTEST.COM for kudu/127.9.66.1@KRBTEST.COM
I20260925 07:32:55.727975 24783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:55.709183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.1:37333 (local address 127.9.66.3:48549)
0925 07:32:55.709259 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0925 07:32:55.709280 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 07:32:55.709354 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:32:55.709656 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:32:55.709661 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:32:55.709937 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:32:55.709996 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:32:55.709998 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:32:55.710841 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:32:55.710842 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:32:55.711425 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:32:55.711427 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:32:55.711969 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:32:55.711970 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:32:55.712058 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:32:55.723831 (+ 11773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:32:55.723835 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:32:55.725185 (+ 1350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:32:55.726949 (+ 1764us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:32:55.726955 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:32:55.726957 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:32:55.727163 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:32:55.727363 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:32:55.727364 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:32:55.727364 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:32:55.727416 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:32:55.727694 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:32:55.727699 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:32:55.727744 (+ 45us) client_negotiation.cc:769] Sending connection context
0925 07:32:55.727809 (+ 65us) client_negotiation.cc:240] Negotiation successful
0925 07:32:55.727896 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260925 07:32:55.727998 24786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:55.709291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.3:48549 (local address 127.9.66.1:37333)
0925 07:32:55.709448 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0925 07:32:55.709452 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 07:32:55.709463 (+ 11us) server_negotiation.cc:407] Connection header received
0925 07:32:55.709489 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:32:55.709490 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:32:55.709565 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:32:55.709644 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:32:55.710072 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:32:55.710745 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:32:55.711521 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:32:55.711857 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:32:55.711931 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:32:55.725265 (+ 13334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:32:55.725274 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:32:55.725274 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:32:55.725281 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:32:55.726867 (+ 1586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:32:55.727241 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:32:55.727243 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:32:55.727244 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:32:55.727304 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:32:55.727472 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:32:55.727473 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:32:55.727473 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:32:55.727635 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:32:55.727705 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 07:32:55.727856 (+ 151us) server_negotiation.cc:299] Negotiation successful
0925 07:32:55.727927 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":88,"threads_started":1}
I20260925 07:32:55.728482 24469 tablet_copy_service.cc:140] P 0b7b4fae549a422797b977bcf5e09cdc: Received BeginTabletCopySession request for tablet ab88041be7cd4c69aa6c46a5890553c9 from peer 74c3871e1dde4f84bda59faeceb249fe ({username='kudu', principal='kudu/127.9.66.3@KRBTEST.COM'} at 127.9.66.3:48549)
I20260925 07:32:55.728575 24469 tablet_copy_service.cc:161] P 0b7b4fae549a422797b977bcf5e09cdc: Beginning new tablet copy session on tablet ab88041be7cd4c69aa6c46a5890553c9 from peer 74c3871e1dde4f84bda59faeceb249fe at {username='kudu', principal='kudu/127.9.66.3@KRBTEST.COM'} at 127.9.66.3:48549: session id = 74c3871e1dde4f84bda59faeceb249fe-ab88041be7cd4c69aa6c46a5890553c9
I20260925 07:32:55.729138 24469 tablet_copy_source_session.cc:205] T ab88041be7cd4c69aa6c46a5890553c9 P 0b7b4fae549a422797b977bcf5e09cdc: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 07:32:55.730237 24784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab88041be7cd4c69aa6c46a5890553c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:32:55.732933 24784 tablet_copy_client.cc:791] T ab88041be7cd4c69aa6c46a5890553c9 P 74c3871e1dde4f84bda59faeceb249fe: tablet copy: Starting download of 0 data blocks...
I20260925 07:32:55.733090 24784 tablet_copy_client.cc:655] T ab88041be7cd4c69aa6c46a5890553c9 P 74c3871e1dde4f84bda59faeceb249fe: tablet copy: Starting download of 1 WAL segments...
I20260925 07:32:55.733899 24784 tablet_copy_client.cc:531] T ab88041be7cd4c69aa6c46a5890553c9 P 74c3871e1dde4f84bda59faeceb249fe: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 07:32:55.734524 24784 tablet_bootstrap.cc:492] T ab88041be7cd4c69aa6c46a5890553c9 P 74c3871e1dde4f84bda59faeceb249fe: Bootstrap starting.
I20260925 07:32:55.735792 24784 log.cc:824] T ab88041be7cd4c69aa6c46a5890553c9 P 74c3871e1dde4f84bda59faeceb249fe: Log is configured to *not* fsync() on all Append() calls
I20260925 07:32:55.738162 24784 tablet_bootstrap.cc:492] T ab88041be7cd4c69aa6c46a5890553c9 P 74c3871e1dde4f84bda59faeceb249fe: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 07:32:55.738407 24784 tablet_bootstrap.cc:492] T ab88041be7cd4c69aa6c46a5890553c9 P 74c3871e1dde4f84bda59faeceb249fe: Bootstrap complete.
I20260925 07:32:55.738554 24784 ts_tablet_manager.cc:1397] T ab88041be7cd4c69aa6c46a5890553c9 P 74c3871e1dde4f84bda59faeceb249fe: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260925 07:32:55.739555 24784 raft_consensus.cc:348] T ab88041be7cd4c69aa6c46a5890553c9 P 74c3871e1dde4f84bda59faeceb249fe [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c59f513dd13b45f4a554c64ff7a03c87" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 38723 } } peers { permanent_uuid: "74c3871e1dde4f84bda59faeceb249fe" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 43023 } } peers { permanent_uuid: "0b7b4fae549a422797b977bcf5e09cdc" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 37333 } }
I20260925 07:32:55.739718 24784 raft_consensus.cc:729] T ab88041be7cd4c69aa6c46a5890553c9 P 74c3871e1dde4f84bda59faeceb249fe [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74c3871e1dde4f84bda59faeceb249fe, State: Initialized, Role: FOLLOWER
I20260925 07:32:55.739863 24784 consensus_queue.cc:260] T ab88041be7cd4c69aa6c46a5890553c9 P 74c3871e1dde4f84bda59faeceb249fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c59f513dd13b45f4a554c64ff7a03c87" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 38723 } } peers { permanent_uuid: "74c3871e1dde4f84bda59faeceb249fe" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 43023 } } peers { permanent_uuid: "0b7b4fae549a422797b977bcf5e09cdc" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 37333 } }
I20260925 07:32:55.740149 24784 ts_tablet_manager.cc:1428] T ab88041be7cd4c69aa6c46a5890553c9 P 74c3871e1dde4f84bda59faeceb249fe: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 07:32:55.740409 24783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:55.740323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:46577 (local address 127.9.66.3:59667)
0925 07:32:55.740355 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0925 07:32:55.740384 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:46577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260925 07:32:55.740412 24469 tablet_copy_service.cc:342] P 0b7b4fae549a422797b977bcf5e09cdc: Request end of tablet copy session 74c3871e1dde4f84bda59faeceb249fe-ab88041be7cd4c69aa6c46a5890553c9 received from {username='kudu', principal='kudu/127.9.66.3@KRBTEST.COM'} at 127.9.66.3:48549
I20260925 07:32:55.740469 24469 tablet_copy_service.cc:433] P 0b7b4fae549a422797b977bcf5e09cdc: ending tablet copy session 74c3871e1dde4f84bda59faeceb249fe-ab88041be7cd4c69aa6c46a5890553c9 on tablet ab88041be7cd4c69aa6c46a5890553c9 with peer 74c3871e1dde4f84bda59faeceb249fe
I20260925 07:32:55.740741 24783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:55.740655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:46577 (local address 127.9.66.3:51277)
0925 07:32:55.740691 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0925 07:32:55.740715 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:46577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260925 07:32:56.124311 24789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:56.124126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:46577 (local address 127.9.66.2:33911)
0925 07:32:56.124231 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0925 07:32:56.124269 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:46577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":66,"threads_started":1}
I20260925 07:32:56.124382 24790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:56.124179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:46577 (local address 127.9.66.1:59379)
0925 07:32:56.124294 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0925 07:32:56.124323 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:46577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":64,"threads_started":1}
I20260925 07:32:56.125697 24790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:56.125598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:46577 (local address 127.9.66.1:51765)
0925 07:32:56.125651 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0925 07:32:56.125677 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:46577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260925 07:32:56.125785 24718 raft_consensus.cc:1203] T ab88041be7cd4c69aa6c46a5890553c9 P 74c3871e1dde4f84bda59faeceb249fe [term 2 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[2.3-2.3] Dedup: 2.3->[]
I20260925 07:32:56.126204 24783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:56.126091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:46577 (local address 127.9.66.3:42719)
0925 07:32:56.126143 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0925 07:32:56.126172 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:46577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260925 07:32:57.124914 24792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:57.124701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:46577 (local address 127.9.66.2:33063)
0925 07:32:57.124833 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0925 07:32:57.124871 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:46577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":54,"threads_started":1}
I20260925 07:32:57.126219 24793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:57.126038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:46577 (local address 127.9.66.1:36027)
0925 07:32:57.126160 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0925 07:32:57.126187 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:46577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":64,"threads_started":1}
I20260925 07:32:57.126806 24794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:57.126625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:46577 (local address 127.9.66.3:46427)
0925 07:32:57.126737 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0925 07:32:57.126772 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:46577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":53,"threads_started":1}
I20260925 07:32:58.125707 24795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:58.125445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:46577 (local address 127.9.66.2:42141)
0925 07:32:58.125625 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0925 07:32:58.125660 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:46577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":84,"threads_started":1}
I20260925 07:32:58.126883 24796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:58.126706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:46577 (local address 127.9.66.1:36045)
0925 07:32:58.126805 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0925 07:32:58.126850 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:46577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":48,"threads_started":1}
I20260925 07:32:58.127367 24797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:58.127203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:46577 (local address 127.9.66.3:59285)
0925 07:32:58.127298 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0925 07:32:58.127331 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:46577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81,"thread_start_us":43,"threads_started":1}
I20260925 07:32:59.126461 24799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:59.126221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:46577 (local address 127.9.66.2:52741)
0925 07:32:59.126373 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 07:32:59.126415 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:46577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":56,"threads_started":1}
I20260925 07:32:59.127501 24800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:59.127309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:46577 (local address 127.9.66.1:55833)
0925 07:32:59.127427 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0925 07:32:59.127460 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:46577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":66,"threads_started":1}
I20260925 07:32:59.127866 24800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:59.127768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:46577 (local address 127.9.66.1:53339)
0925 07:32:59.127815 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0925 07:32:59.127846 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:46577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260925 07:32:59.128031 24801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:32:59.127833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:46577 (local address 127.9.66.3:41433)
0925 07:32:59.127942 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0925 07:32:59.127975 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:46577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":46,"threads_started":1}
I20260925 07:33:00.127170 24802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:33:00.126922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:46577 (local address 127.9.66.2:46499)
0925 07:33:00.127054 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0925 07:33:00.127091 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:46577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":65,"threads_started":1}
I20260925 07:33:00.128458 24803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:33:00.128264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:46577 (local address 127.9.66.1:51361)
0925 07:33:00.128374 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0925 07:33:00.128411 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:46577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":65,"threads_started":1}
I20260925 07:33:00.128635 24804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:33:00.128416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:46577 (local address 127.9.66.3:47615)
0925 07:33:00.128530 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0925 07:33:00.128586 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:46577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":56,"threads_started":1}
I20260925 07:33:01.127871 24805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:33:01.127626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:46577 (local address 127.9.66.2:48575)
0925 07:33:01.127764 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 07:33:01.127803 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:46577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":57,"threads_started":1}
I20260925 07:33:01.129081 24806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:33:01.128894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:46577 (local address 127.9.66.1:57539)
0925 07:33:01.128997 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0925 07:33:01.129036 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:46577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76,"thread_start_us":46,"threads_started":1}
I20260925 07:33:01.129287 24807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:33:01.129040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:46577 (local address 127.9.66.3:37571)
0925 07:33:01.129198 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 07:33:01.129239 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:46577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":60,"threads_started":1}
I20260925 07:33:02.128542 24808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:33:02.128337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:46577 (local address 127.9.66.2:45701)
0925 07:33:02.128460 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0925 07:33:02.128496 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:46577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":62,"threads_started":1}
I20260925 07:33:02.129706 24809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:33:02.129476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:46577 (local address 127.9.66.1:45845)
0925 07:33:02.129638 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 07:33:02.129670 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:46577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":107,"threads_started":1}
I20260925 07:33:02.129906 24810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:33:02.129696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:46577 (local address 127.9.66.3:43559)
0925 07:33:02.129805 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0925 07:33:02.129863 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:46577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":67,"threads_started":1}
I20260925 07:33:03.129173 24811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:33:03.128960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:46577 (local address 127.9.66.2:38849)
0925 07:33:03.129097 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 07:33:03.129127 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:46577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":87,"threads_started":1}
I20260925 07:33:03.130323 24812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:33:03.130114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:46577 (local address 127.9.66.1:41425)
0925 07:33:03.130254 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 07:33:03.130288 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:46577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":70,"threads_started":1}
I20260925 07:33:03.130527 24813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:33:03.130303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:46577 (local address 127.9.66.3:35007)
0925 07:33:03.130411 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0925 07:33:03.130467 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:46577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":69,"threads_started":1}
I20260925 07:33:04.129901 24814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:33:04.129686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:46577 (local address 127.9.66.2:44739)
0925 07:33:04.129817 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0925 07:33:04.129856 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:46577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":82,"threads_started":1}
I20260925 07:33:04.130954 24815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:33:04.130767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:46577 (local address 127.9.66.1:56867)
0925 07:33:04.130881 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0925 07:33:04.130918 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:46577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":72,"threads_started":1}
I20260925 07:33:04.131107 24816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:33:04.130894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:46577 (local address 127.9.66.3:46493)
0925 07:33:04.131006 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0925 07:33:04.131062 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:46577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":70,"threads_started":1}
I20260925 07:33:05.130607 24817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:33:05.130315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:46577 (local address 127.9.66.2:44359)
0925 07:33:05.130517 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0925 07:33:05.130561 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:46577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":71,"threads_started":1}
I20260925 07:33:05.131568 24818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:33:05.131368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:46577 (local address 127.9.66.1:34857)
0925 07:33:05.131502 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0925 07:33:05.131533 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:46577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":50,"threads_started":1}
I20260925 07:33:05.131875 24819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:33:05.131607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:46577 (local address 127.9.66.3:57791)
0925 07:33:05.131758 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 07:33:05.131803 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:46577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":90,"threads_started":1}
I20260925 07:33:06.131459 24820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:33:06.131225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:46577 (local address 127.9.66.2:52857)
0925 07:33:06.131371 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 07:33:06.131409 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:46577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":67,"threads_started":1}
I20260925 07:33:06.132308 24821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:33:06.132026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:46577 (local address 127.9.66.1:39773)
0925 07:33:06.132176 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 07:33:06.132250 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:46577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":77,"threads_started":1}
I20260925 07:33:06.132550 24822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:33:06.132331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:46577 (local address 127.9.66.3:38509)
0925 07:33:06.132461 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 07:33:06.132499 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:46577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":64,"threads_started":1}
I20260925 07:33:07.132325 24823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:33:07.132052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:46577 (local address 127.9.66.2:46457)
0925 07:33:07.132226 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 07:33:07.132277 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:46577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":72,"threads_started":1}
I20260925 07:33:07.132992 24824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:33:07.132780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:46577 (local address 127.9.66.1:49707)
0925 07:33:07.132905 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0925 07:33:07.132954 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:46577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":74,"threads_started":1}
I20260925 07:33:07.133271 24825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:33:07.132993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:46577 (local address 127.9.66.3:49737)
0925 07:33:07.133174 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0925 07:33:07.133212 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:46577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":74,"threads_started":1}
I20260925 07:33:08.133073 24826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:33:08.132877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:46577 (local address 127.9.66.2:33779)
0925 07:33:08.133000 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0925 07:33:08.133030 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:46577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":72,"threads_started":1}
I20260925 07:33:08.133682 24827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:33:08.133418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:46577 (local address 127.9.66.1:50699)
0925 07:33:08.133571 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 07:33:08.133627 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:46577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":48,"threads_started":1}
I20260925 07:33:08.133941 24828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:33:08.133708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:46577 (local address 127.9.66.3:48339)
0925 07:33:08.133863 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 07:33:08.133904 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:46577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":61,"threads_started":1}
I20260925 07:33:08.676041 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 24336
I20260925 07:33:08.680857 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 24499
I20260925 07:33:08.685174 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 24634
2026-09-25T07:33:08Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18474 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 07:33:08 dist-test-slave-57fn krb5kdc[24831](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 07:33:08 dist-test-slave-57fn krb5kdc[24831](info): set up 2 sockets
Sep 25 07:33:08 dist-test-slave-57fn krb5kdc[24831](info): commencing operation
krb5kdc: starting...
W20260925 07:33:09.721431 9480 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/krb5kdc/test-user.keytab.
Sep 25 07:33:09 dist-test-slave-57fn krb5kdc[24831](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321589, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T07:33:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T07:33:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.66.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.66.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.66.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab.
Entry for principal kudu/127.9.66.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab.
I20260925 07:33:09.832512 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.66.62:35239
--webserver_interface=127.9.66.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab
--principal=kudu/127.9.66.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:43203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.66.62:35239
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:33:09.909327 24844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:33:09.909575 24844 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:33:09.909611 24844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:33:09.910748 24844 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 07:33:09.910806 24844 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 07:33:09.910828 24844 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 07:33:09.910857 24844 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:33:09.910884 24844 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:33:09.910912 24844 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:33:09.910939 24844 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:33:09.910967 24844 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:33:09.912377 24844 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:43203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.9.66.62:35239
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab
--principal=kudu/127.9.66.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.66.62:35239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.66.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24844
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:33:09.912643 24844 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:33:09.912850 24844 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:33:09.917367 24852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:33:09.917349 24850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:33:09.917560 24844 server_base.cc:1034] running on GCE node
W20260925 07:33:09.917341 24849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:33:09.917896 24844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:33:09.918233 24844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:33:09.919397 24844 hybrid_clock.cc:648] HybridClock initialized: now 1790321589919387 us; error 29 us; skew 500 ppm
Sep 25 07:33:09 dist-test-slave-57fn krb5kdc[24831](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321589, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:33:09.921772 24844 init.cc:377] Logged in from keytab as kudu/127.9.66.62@KRBTEST.COM (short username kudu)
I20260925 07:33:09.922137 24844 webserver.cc:460] Webserver started at http://127.9.66.62:46119/ using document root <none> and password file <none>
I20260925 07:33:09.922313 24844 fs_manager.cc:359] Metadata directory not provided
I20260925 07:33:09.922351 24844 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:33:09.922430 24844 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:33:09.923357 24844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/master-0/data/instance:
uuid: "10c9b33b17904def90c291c5a7091209"
format_stamp: "Formatted at 2026-09-25 07:33:09 on dist-test-slave-57fn"
server_key: "41bae02c09abafa675bad2414d6b6471"
server_key_iv: "7b491795d817902dbd50efa44281d65e"
server_key_version: "encryptionkey@0"
I20260925 07:33:09.923650 24844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/master-0/wal/instance:
uuid: "10c9b33b17904def90c291c5a7091209"
format_stamp: "Formatted at 2026-09-25 07:33:09 on dist-test-slave-57fn"
server_key: "41bae02c09abafa675bad2414d6b6471"
server_key_iv: "7b491795d817902dbd50efa44281d65e"
server_key_version: "encryptionkey@0"
I20260925 07:33:09.924805 24844 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 07:33:09.925405 24859 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:33:09.925617 24844 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 07:33:09.925676 24844 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/master-0/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/master-0/wal
uuid: "10c9b33b17904def90c291c5a7091209"
format_stamp: "Formatted at 2026-09-25 07:33:09 on dist-test-slave-57fn"
server_key: "41bae02c09abafa675bad2414d6b6471"
server_key_iv: "7b491795d817902dbd50efa44281d65e"
server_key_version: "encryptionkey@0"
I20260925 07:33:09.925724 24844 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:33:09.934579 24844 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:33:09.936182 24844 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:33:09.936342 24844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:33:09.940174 24844 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.62:35239
I20260925 07:33:09.940200 24911 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.62:35239 every 8 connection(s)
I20260925 07:33:09.940523 24844 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
I20260925 07:33:09.941402 24912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:33:09.944345 24912 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10c9b33b17904def90c291c5a7091209: Bootstrap starting.
I20260925 07:33:09.944933 24912 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 10c9b33b17904def90c291c5a7091209: Neither blocks nor log segments found. Creating new log.
I20260925 07:33:09.945222 24912 log.cc:824] T 00000000000000000000000000000000 P 10c9b33b17904def90c291c5a7091209: Log is configured to *not* fsync() on all Append() calls
I20260925 07:33:09.945906 24912 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10c9b33b17904def90c291c5a7091209: No bootstrap required, opened a new log
I20260925 07:33:09.946446 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 24844
I20260925 07:33:09.946548 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/master-0/wal/instance
I20260925 07:33:09.946724 9480 external_mini_cluster.cc:1428] Setting key 6b90ca062381858c5f90f86b67414e5b
I20260925 07:33:09.948007 24912 raft_consensus.cc:348] T 00000000000000000000000000000000 P 10c9b33b17904def90c291c5a7091209 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10c9b33b17904def90c291c5a7091209" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 35239 } }
I20260925 07:33:09.948160 24912 raft_consensus.cc:374] T 00000000000000000000000000000000 P 10c9b33b17904def90c291c5a7091209 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:33:09.948197 24912 raft_consensus.cc:729] T 00000000000000000000000000000000 P 10c9b33b17904def90c291c5a7091209 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10c9b33b17904def90c291c5a7091209, State: Initialized, Role: FOLLOWER
I20260925 07:33:09.948364 24912 consensus_queue.cc:260] T 00000000000000000000000000000000 P 10c9b33b17904def90c291c5a7091209 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10c9b33b17904def90c291c5a7091209" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 35239 } }
I20260925 07:33:09.948457 24912 raft_consensus.cc:388] T 00000000000000000000000000000000 P 10c9b33b17904def90c291c5a7091209 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 07:33:09.948500 24912 raft_consensus.cc:482] T 00000000000000000000000000000000 P 10c9b33b17904def90c291c5a7091209 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 07:33:09.948553 24912 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 10c9b33b17904def90c291c5a7091209 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:33:09.949163 24912 raft_consensus.cc:504] T 00000000000000000000000000000000 P 10c9b33b17904def90c291c5a7091209 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10c9b33b17904def90c291c5a7091209" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 35239 } }
I20260925 07:33:09.949245 24912 leader_election.cc:302] T 00000000000000000000000000000000 P 10c9b33b17904def90c291c5a7091209 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 10c9b33b17904def90c291c5a7091209; no voters:
I20260925 07:33:09.949453 24912 leader_election.cc:288] T 00000000000000000000000000000000 P 10c9b33b17904def90c291c5a7091209 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 07:33:09.949477 24917 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 10c9b33b17904def90c291c5a7091209 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:33:09.949602 24917 raft_consensus.cc:686] T 00000000000000000000000000000000 P 10c9b33b17904def90c291c5a7091209 [term 1 LEADER]: Becoming Leader. State: Replica: 10c9b33b17904def90c291c5a7091209, State: Running, Role: LEADER
I20260925 07:33:09.949731 24912 sys_catalog.cc:564] T 00000000000000000000000000000000 P 10c9b33b17904def90c291c5a7091209 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:33:09.949738 24917 consensus_queue.cc:237] T 00000000000000000000000000000000 P 10c9b33b17904def90c291c5a7091209 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10c9b33b17904def90c291c5a7091209" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 35239 } }
I20260925 07:33:09.950157 24919 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10c9b33b17904def90c291c5a7091209 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 10c9b33b17904def90c291c5a7091209. Latest consensus state: current_term: 1 leader_uuid: "10c9b33b17904def90c291c5a7091209" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10c9b33b17904def90c291c5a7091209" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 35239 } } }
I20260925 07:33:09.950266 24919 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10c9b33b17904def90c291c5a7091209 [sys.catalog]: This master's current role is: LEADER
I20260925 07:33:09.950173 24918 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10c9b33b17904def90c291c5a7091209 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "10c9b33b17904def90c291c5a7091209" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10c9b33b17904def90c291c5a7091209" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 35239 } } }
I20260925 07:33:09.950397 24918 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10c9b33b17904def90c291c5a7091209 [sys.catalog]: This master's current role is: LEADER
I20260925 07:33:09.950677 24924 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 07:33:09.951208 24924 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 07:33:09.953634 24924 catalog_manager.cc:1347] Generated new cluster ID: dd1c0386aa0144abaed6dc16b9d4fffc
I20260925 07:33:09.953686 24924 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 07:33:09 dist-test-slave-57fn krb5kdc[24831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321589, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.66.62@KRBTEST.COM
I20260925 07:33:09.958519 24916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:33:09.947225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46292 (local address 127.9.66.62:35239)
0925 07:33:09.947398 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 07:33:09.947405 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 07:33:09.947429 (+ 24us) server_negotiation.cc:407] Connection header received
0925 07:33:09.947990 (+ 561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:33:09.947996 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:33:09.948038 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:33:09.948143 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:33:09.948915 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:33:09.950486 (+ 1571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:33:09.951584 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:33:09.952084 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:33:09.952205 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:33:09.954653 (+ 2448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:33:09.954656 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:33:09.954657 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:33:09.954667 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:33:09.956919 (+ 2252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:33:09.957237 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:33:09.957239 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:33:09.957239 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:33:09.957326 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:33:09.957472 (+ 146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:33:09.957472 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:33:09.957472 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:33:09.957724 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:33:09.957839 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 07:33:09.958229 (+ 390us) server_negotiation.cc:299] Negotiation successful
0925 07:33:09.958337 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":88,"threads_started":1}
I20260925 07:33:09.970714 24924 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 07:33:09.971397 24924 catalog_manager.cc:1504] Loading token signing keys...
I20260925 07:33:09.975006 24924 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 10c9b33b17904def90c291c5a7091209: Generated new TSK 0
I20260925 07:33:09.975184 24924 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.66.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.66.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.66.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.1.keytab.
Entry for principal kudu/127.9.66.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.1.keytab.
I20260925 07:33:10.004029 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.66.1:0
--local_ip_for_outbound_sockets=127.9.66.1
--webserver_interface=127.9.66.1
--webserver_port=0
--tserver_master_addrs=127.9.66.62:35239
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.1.keytab
--principal=kudu/127.9.66.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:43203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:33:10.083026 24938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:33:10.083259 24938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:33:10.083299 24938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:33:10.084410 24938 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:33:10.084481 24938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:33:10.084527 24938 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.66.1
I20260925 07:33:10.086040 24938 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:43203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.1.keytab
--principal=kudu/127.9.66.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.66.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.66.1
--webserver_port=0
--tserver_master_addrs=127.9.66.62:35239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24938
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.66.1
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:33:10.086287 24938 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:33:10.086489 24938 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:33:10.091256 24946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:33:10.091264 24943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:33:10.091292 24944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:33:10.091696 24938 server_base.cc:1034] running on GCE node
I20260925 07:33:10.091856 24938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:33:10.092056 24938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:33:10.093187 24938 hybrid_clock.cc:648] HybridClock initialized: now 1790321590093159 us; error 45 us; skew 500 ppm
Sep 25 07:33:10 dist-test-slave-57fn krb5kdc[24831](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321590, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:33:10.095496 24938 init.cc:377] Logged in from keytab as kudu/127.9.66.1@KRBTEST.COM (short username kudu)
I20260925 07:33:10.095873 24938 webserver.cc:460] Webserver started at http://127.9.66.1:32927/ using document root <none> and password file <none>
I20260925 07:33:10.096087 24938 fs_manager.cc:359] Metadata directory not provided
I20260925 07:33:10.096148 24938 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:33:10.096287 24938 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:33:10.097265 24938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-0/data/instance:
uuid: "fb10411db4a0426fa59c5f405fbcaf03"
format_stamp: "Formatted at 2026-09-25 07:33:10 on dist-test-slave-57fn"
server_key: "896b5afaa9608339fc8459e883be320b"
server_key_iv: "5adf2d1d9de966513d2a996252efe614"
server_key_version: "encryptionkey@0"
I20260925 07:33:10.097649 24938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-0/wal/instance:
uuid: "fb10411db4a0426fa59c5f405fbcaf03"
format_stamp: "Formatted at 2026-09-25 07:33:10 on dist-test-slave-57fn"
server_key: "896b5afaa9608339fc8459e883be320b"
server_key_iv: "5adf2d1d9de966513d2a996252efe614"
server_key_version: "encryptionkey@0"
I20260925 07:33:10.098937 24938 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 07:33:10.099602 24953 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:33:10.099771 24938 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 07:33:10.099859 24938 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-0/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-0/wal
uuid: "fb10411db4a0426fa59c5f405fbcaf03"
format_stamp: "Formatted at 2026-09-25 07:33:10 on dist-test-slave-57fn"
server_key: "896b5afaa9608339fc8459e883be320b"
server_key_iv: "5adf2d1d9de966513d2a996252efe614"
server_key_version: "encryptionkey@0"
I20260925 07:33:10.099936 24938 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:33:10.111199 24938 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:33:10.112864 24938 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:33:10.113059 24938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:33:10.113267 24938 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:33:10.113617 24938 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:33:10.113674 24938 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:33:10.113708 24938 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:33:10.113734 24938 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:33:10.119637 24938 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.1:34439
I20260925 07:33:10.119673 25066 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.1:34439 every 8 connection(s)
I20260925 07:33:10.120020 24938 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-0/data/info.pb
Sep 25 07:33:10 dist-test-slave-57fn krb5kdc[24831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321590, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.1@KRBTEST.COM for kudu/127.9.66.62@KRBTEST.COM
I20260925 07:33:10.128721 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 24938
I20260925 07:33:10.128834 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-0/wal/instance
I20260925 07:33:10.128938 24916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:33:10.120725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.1:43439 (local address 127.9.66.62:35239)
0925 07:33:10.120811 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0925 07:33:10.120814 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:33:10.120979 (+ 165us) server_negotiation.cc:407] Connection header received
0925 07:33:10.121417 (+ 438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:33:10.121418 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:33:10.121428 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:33:10.121496 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:33:10.122482 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:33:10.122985 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:33:10.123927 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:33:10.124159 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:33:10.124225 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:33:10.126343 (+ 2118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:33:10.126345 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:33:10.126345 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:33:10.126357 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:33:10.127636 (+ 1279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:33:10.128067 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:33:10.128068 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:33:10.128068 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:33:10.128118 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:33:10.128316 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:33:10.128316 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:33:10.128316 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:33:10.128408 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:33:10.128466 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 07:33:10.128715 (+ 249us) server_negotiation.cc:299] Negotiation successful
0925 07:33:10.128807 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 07:33:10.129065 9480 external_mini_cluster.cc:1428] Setting key a34170d0834aa913d6ae73c2a9941821
I20260925 07:33:10.129068 25069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:33:10.120685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:35239 (local address 127.9.66.1:43439)
0925 07:33:10.120855 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 07:33:10.120888 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 07:33:10.121337 (+ 449us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:33:10.121615 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:33:10.121618 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:33:10.122181 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:33:10.122381 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:33:10.122387 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:33:10.123058 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:33:10.123058 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:33:10.123827 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:33:10.123829 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:33:10.124239 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:33:10.124239 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:33:10.124332 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:33:10.124844 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:33:10.124853 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:33:10.126240 (+ 1387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:33:10.127708 (+ 1468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:33:10.127714 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:33:10.127715 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:33:10.128007 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:33:10.128179 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:33:10.128179 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:33:10.128180 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:33:10.128273 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:33:10.128456 (+ 183us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:33:10.128460 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:33:10.128586 (+ 126us) client_negotiation.cc:769] Sending connection context
0925 07:33:10.128665 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 07:33:10.128780 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":89,"threads_started":1}
I20260925 07:33:10.129678 25067 heartbeater.cc:347] Connected to a master server at 127.9.66.62:35239
I20260925 07:33:10.129789 25067 heartbeater.cc:464] Registering TS with master...
I20260925 07:33:10.129956 25067 heartbeater.cc:511] Master 127.9.66.62:35239 requested a full tablet report, sending...
I20260925 07:33:10.130481 24876 ts_manager.cc:194] Registered new tserver with Master: fb10411db4a0426fa59c5f405fbcaf03 (127.9.66.1:34439)
I20260925 07:33:10.131078 24876 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.66.1@KRBTEST.COM'} at 127.9.66.1:43439
WARNING: no policy specified for kudu/127.9.66.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.66.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.66.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.2.keytab.
Entry for principal kudu/127.9.66.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.2.keytab.
I20260925 07:33:10.152850 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.66.2:0
--local_ip_for_outbound_sockets=127.9.66.2
--webserver_interface=127.9.66.2
--webserver_port=0
--tserver_master_addrs=127.9.66.62:35239
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.2.keytab
--principal=kudu/127.9.66.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:43203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:33:10.230938 25072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:33:10.231150 25072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:33:10.231184 25072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:33:10.232326 25072 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:33:10.232409 25072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:33:10.232455 25072 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.66.2
I20260925 07:33:10.233958 25072 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:43203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.2.keytab
--principal=kudu/127.9.66.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.66.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.66.2
--webserver_port=0
--tserver_master_addrs=127.9.66.62:35239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25072
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.66.2
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:33:10.234210 25072 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:33:10.234437 25072 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:33:10.239462 25078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:33:10.239477 25077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:33:10.239513 25080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:33:10.239717 25072 server_base.cc:1034] running on GCE node
I20260925 07:33:10.239923 25072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:33:10.240156 25072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:33:10.241341 25072 hybrid_clock.cc:648] HybridClock initialized: now 1790321590241318 us; error 32 us; skew 500 ppm
Sep 25 07:33:10 dist-test-slave-57fn krb5kdc[24831](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321590, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:33:10.243690 25072 init.cc:377] Logged in from keytab as kudu/127.9.66.2@KRBTEST.COM (short username kudu)
I20260925 07:33:10.244148 25072 webserver.cc:460] Webserver started at http://127.9.66.2:34779/ using document root <none> and password file <none>
I20260925 07:33:10.244412 25072 fs_manager.cc:359] Metadata directory not provided
I20260925 07:33:10.244481 25072 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:33:10.244632 25072 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:33:10.245721 25072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-1/data/instance:
uuid: "a3e2cf0a23c14e3e876406afcfdd6e07"
format_stamp: "Formatted at 2026-09-25 07:33:10 on dist-test-slave-57fn"
server_key: "249ac6722ef820f6a1335e766d83a839"
server_key_iv: "42dc75f8fcd51bab6b2ab4736611a7c7"
server_key_version: "encryptionkey@0"
I20260925 07:33:10.246068 25072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-1/wal/instance:
uuid: "a3e2cf0a23c14e3e876406afcfdd6e07"
format_stamp: "Formatted at 2026-09-25 07:33:10 on dist-test-slave-57fn"
server_key: "249ac6722ef820f6a1335e766d83a839"
server_key_iv: "42dc75f8fcd51bab6b2ab4736611a7c7"
server_key_version: "encryptionkey@0"
I20260925 07:33:10.247335 25072 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:33:10.248030 25087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:33:10.248202 25072 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 07:33:10.248299 25072 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-1/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-1/wal
uuid: "a3e2cf0a23c14e3e876406afcfdd6e07"
format_stamp: "Formatted at 2026-09-25 07:33:10 on dist-test-slave-57fn"
server_key: "249ac6722ef820f6a1335e766d83a839"
server_key_iv: "42dc75f8fcd51bab6b2ab4736611a7c7"
server_key_version: "encryptionkey@0"
I20260925 07:33:10.248373 25072 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:33:10.261818 25072 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:33:10.263491 25072 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:33:10.263639 25072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:33:10.263839 25072 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:33:10.264119 25072 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:33:10.264170 25072 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:33:10.264209 25072 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:33:10.264236 25072 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:33:10.270150 25072 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.2:33299
I20260925 07:33:10.270201 25200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.2:33299 every 8 connection(s)
I20260925 07:33:10.270485 25072 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-1/data/info.pb
Sep 25 07:33:10 dist-test-slave-57fn krb5kdc[24831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321590, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.2@KRBTEST.COM for kudu/127.9.66.62@KRBTEST.COM
I20260925 07:33:10.277268 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 25072
I20260925 07:33:10.277354 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-1/wal/instance
I20260925 07:33:10.277598 9480 external_mini_cluster.cc:1428] Setting key 0eb0ec5804d20adc8b19745c47a98213
I20260925 07:33:10.279232 24916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:33:10.271153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.2:42365 (local address 127.9.66.62:35239)
0925 07:33:10.271215 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 07:33:10.271218 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:33:10.271425 (+ 207us) server_negotiation.cc:407] Connection header received
0925 07:33:10.271852 (+ 427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:33:10.271853 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:33:10.271862 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:33:10.271924 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:33:10.272745 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:33:10.273226 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:33:10.274146 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:33:10.274334 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:33:10.274382 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:33:10.276323 (+ 1941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:33:10.276325 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:33:10.276325 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:33:10.276348 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:33:10.277839 (+ 1491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:33:10.278288 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:33:10.278289 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:33:10.278290 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:33:10.278352 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:33:10.278596 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:33:10.278597 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:33:10.278597 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:33:10.278709 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:33:10.278781 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 07:33:10.279060 (+ 279us) server_negotiation.cc:299] Negotiation successful
0925 07:33:10.279142 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 07:33:10.279433 25203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:33:10.271120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:35239 (local address 127.9.66.2:42365)
0925 07:33:10.271298 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0925 07:33:10.271336 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 07:33:10.271777 (+ 441us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:33:10.271973 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:33:10.271976 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:33:10.272503 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:33:10.272674 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:33:10.272678 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:33:10.273304 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:33:10.273305 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:33:10.274079 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:33:10.274081 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:33:10.274399 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:33:10.274399 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:33:10.274492 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:33:10.274961 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:33:10.274969 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:33:10.276249 (+ 1280us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:33:10.277920 (+ 1671us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:33:10.277926 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:33:10.277928 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:33:10.278218 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:33:10.278420 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:33:10.278421 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:33:10.278421 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:33:10.278529 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:33:10.278774 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:33:10.278779 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:33:10.278914 (+ 135us) client_negotiation.cc:769] Sending connection context
0925 07:33:10.279039 (+ 125us) client_negotiation.cc:240] Negotiation successful
0925 07:33:10.279190 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":95,"threads_started":1}
I20260925 07:33:10.280028 25201 heartbeater.cc:347] Connected to a master server at 127.9.66.62:35239
I20260925 07:33:10.280126 25201 heartbeater.cc:464] Registering TS with master...
I20260925 07:33:10.280305 25201 heartbeater.cc:511] Master 127.9.66.62:35239 requested a full tablet report, sending...
I20260925 07:33:10.280674 24876 ts_manager.cc:194] Registered new tserver with Master: a3e2cf0a23c14e3e876406afcfdd6e07 (127.9.66.2:33299)
I20260925 07:33:10.281178 24876 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.66.2@KRBTEST.COM'} at 127.9.66.2:42365
WARNING: no policy specified for kudu/127.9.66.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.66.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.66.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.3.keytab.
Entry for principal kudu/127.9.66.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.3.keytab.
I20260925 07:33:10.302076 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.66.3:0
--local_ip_for_outbound_sockets=127.9.66.3
--webserver_interface=127.9.66.3
--webserver_port=0
--tserver_master_addrs=127.9.66.62:35239
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.3.keytab
--principal=kudu/127.9.66.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:43203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:33:10.381472 25206 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:33:10.381675 25206 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:33:10.381697 25206 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:33:10.382761 25206 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:33:10.382793 25206 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:33:10.382817 25206 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.66.3
I20260925 07:33:10.384143 25206 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:43203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.3.keytab
--principal=kudu/127.9.66.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.66.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.66.3
--webserver_port=0
--tserver_master_addrs=127.9.66.62:35239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25206
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.66.3
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:33:10.384310 25206 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:33:10.384505 25206 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:33:10.389443 25211 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:33:10.389454 25214 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:33:10.389479 25212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:33:10.389639 25206 server_base.cc:1034] running on GCE node
I20260925 07:33:10.389808 25206 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:33:10.389982 25206 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:33:10.391111 25206 hybrid_clock.cc:648] HybridClock initialized: now 1790321590391098 us; error 32 us; skew 500 ppm
Sep 25 07:33:10 dist-test-slave-57fn krb5kdc[24831](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321590, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:33:10.393679 25206 init.cc:377] Logged in from keytab as kudu/127.9.66.3@KRBTEST.COM (short username kudu)
I20260925 07:33:10.394116 25206 webserver.cc:460] Webserver started at http://127.9.66.3:40087/ using document root <none> and password file <none>
I20260925 07:33:10.394321 25206 fs_manager.cc:359] Metadata directory not provided
I20260925 07:33:10.394363 25206 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:33:10.394521 25206 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:33:10.395481 25206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-2/data/instance:
uuid: "693a2a3014a34660ad9c23b17abceed7"
format_stamp: "Formatted at 2026-09-25 07:33:10 on dist-test-slave-57fn"
server_key: "40e264dce66644c9faa1b5b316241f02"
server_key_iv: "d0fb262afd73437e77767ec549298845"
server_key_version: "encryptionkey@0"
I20260925 07:33:10.395819 25206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-2/wal/instance:
uuid: "693a2a3014a34660ad9c23b17abceed7"
format_stamp: "Formatted at 2026-09-25 07:33:10 on dist-test-slave-57fn"
server_key: "40e264dce66644c9faa1b5b316241f02"
server_key_iv: "d0fb262afd73437e77767ec549298845"
server_key_version: "encryptionkey@0"
I20260925 07:33:10.397013 25206 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 07:33:10.397858 25221 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:33:10.398028 25206 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 07:33:10.398109 25206 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-2/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-2/wal
uuid: "693a2a3014a34660ad9c23b17abceed7"
format_stamp: "Formatted at 2026-09-25 07:33:10 on dist-test-slave-57fn"
server_key: "40e264dce66644c9faa1b5b316241f02"
server_key_iv: "d0fb262afd73437e77767ec549298845"
server_key_version: "encryptionkey@0"
I20260925 07:33:10.398180 25206 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:33:10.410373 25206 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:33:10.412554 25206 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:33:10.412704 25206 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:33:10.412917 25206 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:33:10.413205 25206 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:33:10.413237 25206 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:33:10.413259 25206 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:33:10.413300 25206 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:33:10.419196 25206 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.3:46457
I20260925 07:33:10.419231 25334 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.3:46457 every 8 connection(s)
I20260925 07:33:10.419552 25206 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-2/data/info.pb
Sep 25 07:33:10 dist-test-slave-57fn krb5kdc[24831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321590, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.3@KRBTEST.COM for kudu/127.9.66.62@KRBTEST.COM
I20260925 07:33:10.426484 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 25206
I20260925 07:33:10.426589 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-2/wal/instance
I20260925 07:33:10.426797 9480 external_mini_cluster.cc:1428] Setting key 6ac84ef6cc4c6ee3d08b9f993c0e3528
I20260925 07:33:10.428138 24916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:33:10.420232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.3:55045 (local address 127.9.66.62:35239)
0925 07:33:10.420271 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 07:33:10.420273 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:33:10.420465 (+ 192us) server_negotiation.cc:407] Connection header received
0925 07:33:10.420896 (+ 431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:33:10.420897 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:33:10.420906 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:33:10.420956 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:33:10.421889 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:33:10.422379 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:33:10.423281 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:33:10.423457 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:33:10.423510 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:33:10.425320 (+ 1810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:33:10.425321 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:33:10.425322 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:33:10.425328 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:33:10.426725 (+ 1397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:33:10.427173 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:33:10.427175 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:33:10.427175 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:33:10.427237 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:33:10.427537 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:33:10.427538 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:33:10.427538 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:33:10.427694 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:33:10.427765 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 07:33:10.428014 (+ 249us) server_negotiation.cc:299] Negotiation successful
0925 07:33:10.428077 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 07:33:10.428294 25337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:33:10.420214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:35239 (local address 127.9.66.3:55045)
0925 07:33:10.420347 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0925 07:33:10.420379 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0925 07:33:10.420819 (+ 440us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:33:10.421010 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:33:10.421013 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:33:10.421632 (+ 619us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:33:10.421814 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:33:10.421818 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:33:10.422455 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:33:10.422456 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:33:10.423210 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:33:10.423212 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:33:10.423530 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:33:10.423531 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:33:10.423623 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:33:10.424004 (+ 381us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:33:10.424011 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:33:10.425244 (+ 1233us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:33:10.426799 (+ 1555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:33:10.426804 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:33:10.426806 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:33:10.427081 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:33:10.427339 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:33:10.427340 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:33:10.427340 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:33:10.427436 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:33:10.427763 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:33:10.427768 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:33:10.427894 (+ 126us) client_negotiation.cc:769] Sending connection context
0925 07:33:10.427975 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 07:33:10.428089 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":78,"threads_started":1}
I20260925 07:33:10.428784 25335 heartbeater.cc:347] Connected to a master server at 127.9.66.62:35239
I20260925 07:33:10.428870 25335 heartbeater.cc:464] Registering TS with master...
I20260925 07:33:10.429003 25335 heartbeater.cc:511] Master 127.9.66.62:35239 requested a full tablet report, sending...
I20260925 07:33:10.429319 24876 ts_manager.cc:194] Registered new tserver with Master: 693a2a3014a34660ad9c23b17abceed7 (127.9.66.3:46457)
I20260925 07:33:10.429704 24876 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.66.3@KRBTEST.COM'} at 127.9.66.3:55045
I20260925 07:33:10.438640 9480 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 07:33:10.445212 24916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:33:10.439961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46308 (local address 127.9.66.62:35239)
0925 07:33:10.440025 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 07:33:10.440027 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:33:10.440120 (+ 93us) server_negotiation.cc:407] Connection header received
0925 07:33:10.440141 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:33:10.440142 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:33:10.440150 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:33:10.440196 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:33:10.440724 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:33:10.441175 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:33:10.441898 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:33:10.442072 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:33:10.442120 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:33:10.443210 (+ 1090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:33:10.443211 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:33:10.443211 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:33:10.443217 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:33:10.444314 (+ 1097us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:33:10.444621 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:33:10.444621 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:33:10.444622 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:33:10.444666 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:33:10.444811 (+ 145us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:33:10.444811 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:33:10.444812 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:33:10.444921 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:33:10.444992 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 07:33:10.445080 (+ 88us) server_negotiation.cc:299] Negotiation successful
0925 07:33:10.445133 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 07:33:10.446750 24876 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46308:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 07:33:10.448289 24876 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 07:33:10.454736 25345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:33:10.450688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.3:46457 (local address 127.0.0.1:35266)
0925 07:33:10.450764 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0925 07:33:10.450774 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 07:33:10.450820 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:33:10.451350 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:33:10.451351 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:33:10.451352 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:33:10.451395 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:33:10.451397 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:33:10.452648 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:33:10.452649 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:33:10.453783 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:33:10.453785 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:33:10.454455 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:33:10.454456 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:33:10.454530 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:33:10.454543 (+ 13us) client_negotiation.cc:769] Sending connection context
0925 07:33:10.454614 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 07:33:10.454657 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62,"thread_start_us":44,"threads_started":1}
I20260925 07:33:10.454766 25348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:33:10.451001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35266 (local address 127.9.66.3:46457)
0925 07:33:10.451208 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0925 07:33:10.451214 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 07:33:10.451223 (+ 9us) server_negotiation.cc:407] Connection header received
0925 07:33:10.451259 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:33:10.451261 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:33:10.451274 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:33:10.451351 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:33:10.451608 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:33:10.452565 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:33:10.453867 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:33:10.454366 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:33:10.454432 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:33:10.454468 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0925 07:33:10.454645 (+ 177us) server_negotiation.cc:299] Negotiation successful
0925 07:33:10.454685 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":83,"threads_started":1}
I20260925 07:33:10.455593 25343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:33:10.450449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.2:33299 (local address 127.0.0.1:38064)
0925 07:33:10.450589 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 07:33:10.450633 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 07:33:10.450713 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:33:10.450991 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:33:10.450994 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:33:10.450996 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:33:10.451063 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:33:10.451065 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:33:10.452396 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:33:10.452397 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:33:10.453792 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:33:10.453794 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:33:10.455363 (+ 1569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:33:10.455364 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:33:10.455426 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:33:10.455429 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 07:33:10.455485 (+ 56us) client_negotiation.cc:240] Negotiation successful
0925 07:33:10.455524 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"mutex_wait_us":33,"thread_start_us":97,"threads_started":1}
I20260925 07:33:10.455476 25269 tablet_service.cc:1467] Processing CreateTablet for tablet 8fd7878e0398470aa866f1b32cd324ec (DEFAULT_TABLE table=test-table [id=b9b78c1e10a94a52bdc4bc3e3baf439f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:33:10.455614 25344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:33:10.450577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38064 (local address 127.9.66.2:33299)
0925 07:33:10.450750 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 07:33:10.450755 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 07:33:10.450765 (+ 10us) server_negotiation.cc:407] Connection header received
0925 07:33:10.450804 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:33:10.450806 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:33:10.450821 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:33:10.450898 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:33:10.451150 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:33:10.452303 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:33:10.454505 (+ 2202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:33:10.455261 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:33:10.455358 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:33:10.455410 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 07:33:10.455505 (+ 95us) server_negotiation.cc:299] Negotiation successful
0925 07:33:10.455540 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":78,"threads_started":1}
I20260925 07:33:10.455835 25269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fd7878e0398470aa866f1b32cd324ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:33:10.455994 25346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:33:10.450834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.1:34439 (local address 127.0.0.1:35504)
0925 07:33:10.451056 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0925 07:33:10.451067 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 07:33:10.451114 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:33:10.452838 (+ 1724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:33:10.452839 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:33:10.452841 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:33:10.452890 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:33:10.452892 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:33:10.453920 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:33:10.453921 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:33:10.454703 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:33:10.454705 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:33:10.455741 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:33:10.455742 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:33:10.455818 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:33:10.455822 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 07:33:10.455881 (+ 59us) client_negotiation.cc:240] Negotiation successful
0925 07:33:10.455918 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":37,"threads_started":1}
I20260925 07:33:10.456113 25347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:33:10.450955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35504 (local address 127.9.66.1:34439)
0925 07:33:10.452698 (+ 1743us) server_negotiation.cc:206] Beginning negotiation
0925 07:33:10.452701 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:33:10.452709 (+ 8us) server_negotiation.cc:407] Connection header received
0925 07:33:10.452738 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:33:10.452739 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:33:10.452751 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:33:10.452839 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:33:10.452946 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:33:10.453809 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:33:10.454791 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:33:10.455587 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:33:10.455655 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:33:10.455705 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 07:33:10.455925 (+ 220us) server_negotiation.cc:299] Negotiation successful
0925 07:33:10.455968 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1711,"thread_start_us":90,"threads_started":1}
I20260925 07:33:10.456344 25135 tablet_service.cc:1467] Processing CreateTablet for tablet 8fd7878e0398470aa866f1b32cd324ec (DEFAULT_TABLE table=test-table [id=b9b78c1e10a94a52bdc4bc3e3baf439f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:33:10.456574 25135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fd7878e0398470aa866f1b32cd324ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:33:10.458498 25349 tablet_bootstrap.cc:492] T 8fd7878e0398470aa866f1b32cd324ec P 693a2a3014a34660ad9c23b17abceed7: Bootstrap starting.
I20260925 07:33:10.458968 25350 tablet_bootstrap.cc:492] T 8fd7878e0398470aa866f1b32cd324ec P a3e2cf0a23c14e3e876406afcfdd6e07: Bootstrap starting.
I20260925 07:33:10.459270 25349 tablet_bootstrap.cc:654] T 8fd7878e0398470aa866f1b32cd324ec P 693a2a3014a34660ad9c23b17abceed7: Neither blocks nor log segments found. Creating new log.
I20260925 07:33:10.459275 25001 tablet_service.cc:1467] Processing CreateTablet for tablet 8fd7878e0398470aa866f1b32cd324ec (DEFAULT_TABLE table=test-table [id=b9b78c1e10a94a52bdc4bc3e3baf439f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:33:10.459594 25350 tablet_bootstrap.cc:654] T 8fd7878e0398470aa866f1b32cd324ec P a3e2cf0a23c14e3e876406afcfdd6e07: Neither blocks nor log segments found. Creating new log.
I20260925 07:33:10.459627 25001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fd7878e0398470aa866f1b32cd324ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:33:10.459901 25349 log.cc:824] T 8fd7878e0398470aa866f1b32cd324ec P 693a2a3014a34660ad9c23b17abceed7: Log is configured to *not* fsync() on all Append() calls
I20260925 07:33:10.460139 25350 log.cc:824] T 8fd7878e0398470aa866f1b32cd324ec P a3e2cf0a23c14e3e876406afcfdd6e07: Log is configured to *not* fsync() on all Append() calls
I20260925 07:33:10.460634 25349 tablet_bootstrap.cc:492] T 8fd7878e0398470aa866f1b32cd324ec P 693a2a3014a34660ad9c23b17abceed7: No bootstrap required, opened a new log
I20260925 07:33:10.460709 25349 ts_tablet_manager.cc:1397] T 8fd7878e0398470aa866f1b32cd324ec P 693a2a3014a34660ad9c23b17abceed7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 07:33:10.460894 25350 tablet_bootstrap.cc:492] T 8fd7878e0398470aa866f1b32cd324ec P a3e2cf0a23c14e3e876406afcfdd6e07: No bootstrap required, opened a new log
I20260925 07:33:10.460955 25350 ts_tablet_manager.cc:1397] T 8fd7878e0398470aa866f1b32cd324ec P a3e2cf0a23c14e3e876406afcfdd6e07: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 07:33:10.462611 25353 tablet_bootstrap.cc:492] T 8fd7878e0398470aa866f1b32cd324ec P fb10411db4a0426fa59c5f405fbcaf03: Bootstrap starting.
I20260925 07:33:10.462668 25349 raft_consensus.cc:348] T 8fd7878e0398470aa866f1b32cd324ec P 693a2a3014a34660ad9c23b17abceed7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e2cf0a23c14e3e876406afcfdd6e07" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 33299 } } peers { permanent_uuid: "693a2a3014a34660ad9c23b17abceed7" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 46457 } } peers { permanent_uuid: "fb10411db4a0426fa59c5f405fbcaf03" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 34439 } }
I20260925 07:33:10.462852 25349 raft_consensus.cc:374] T 8fd7878e0398470aa866f1b32cd324ec P 693a2a3014a34660ad9c23b17abceed7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:33:10.462904 25349 raft_consensus.cc:729] T 8fd7878e0398470aa866f1b32cd324ec P 693a2a3014a34660ad9c23b17abceed7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 693a2a3014a34660ad9c23b17abceed7, State: Initialized, Role: FOLLOWER
I20260925 07:33:10.462873 25350 raft_consensus.cc:348] T 8fd7878e0398470aa866f1b32cd324ec P a3e2cf0a23c14e3e876406afcfdd6e07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e2cf0a23c14e3e876406afcfdd6e07" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 33299 } } peers { permanent_uuid: "693a2a3014a34660ad9c23b17abceed7" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 46457 } } peers { permanent_uuid: "fb10411db4a0426fa59c5f405fbcaf03" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 34439 } }
I20260925 07:33:10.463025 25349 consensus_queue.cc:260] T 8fd7878e0398470aa866f1b32cd324ec P 693a2a3014a34660ad9c23b17abceed7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e2cf0a23c14e3e876406afcfdd6e07" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 33299 } } peers { permanent_uuid: "693a2a3014a34660ad9c23b17abceed7" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 46457 } } peers { permanent_uuid: "fb10411db4a0426fa59c5f405fbcaf03" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 34439 } }
I20260925 07:33:10.463074 25350 raft_consensus.cc:374] T 8fd7878e0398470aa866f1b32cd324ec P a3e2cf0a23c14e3e876406afcfdd6e07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:33:10.463109 25350 raft_consensus.cc:729] T 8fd7878e0398470aa866f1b32cd324ec P a3e2cf0a23c14e3e876406afcfdd6e07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3e2cf0a23c14e3e876406afcfdd6e07, State: Initialized, Role: FOLLOWER
I20260925 07:33:10.463238 25350 consensus_queue.cc:260] T 8fd7878e0398470aa866f1b32cd324ec P a3e2cf0a23c14e3e876406afcfdd6e07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e2cf0a23c14e3e876406afcfdd6e07" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 33299 } } peers { permanent_uuid: "693a2a3014a34660ad9c23b17abceed7" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 46457 } } peers { permanent_uuid: "fb10411db4a0426fa59c5f405fbcaf03" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 34439 } }
I20260925 07:33:10.463291 25349 ts_tablet_manager.cc:1428] T 8fd7878e0398470aa866f1b32cd324ec P 693a2a3014a34660ad9c23b17abceed7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 07:33:10.463320 25353 tablet_bootstrap.cc:654] T 8fd7878e0398470aa866f1b32cd324ec P fb10411db4a0426fa59c5f405fbcaf03: Neither blocks nor log segments found. Creating new log.
I20260925 07:33:10.463315 25335 heartbeater.cc:503] Master 127.9.66.62:35239 was elected leader, sending a full tablet report...
I20260925 07:33:10.463508 25201 heartbeater.cc:503] Master 127.9.66.62:35239 was elected leader, sending a full tablet report...
I20260925 07:33:10.463498 25350 ts_tablet_manager.cc:1428] T 8fd7878e0398470aa866f1b32cd324ec P a3e2cf0a23c14e3e876406afcfdd6e07: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 07:33:10.464195 25353 log.cc:824] T 8fd7878e0398470aa866f1b32cd324ec P fb10411db4a0426fa59c5f405fbcaf03: Log is configured to *not* fsync() on all Append() calls
I20260925 07:33:10.465000 25353 tablet_bootstrap.cc:492] T 8fd7878e0398470aa866f1b32cd324ec P fb10411db4a0426fa59c5f405fbcaf03: No bootstrap required, opened a new log
I20260925 07:33:10.465066 25353 ts_tablet_manager.cc:1397] T 8fd7878e0398470aa866f1b32cd324ec P fb10411db4a0426fa59c5f405fbcaf03: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 07:33:10.466380 25353 raft_consensus.cc:348] T 8fd7878e0398470aa866f1b32cd324ec P fb10411db4a0426fa59c5f405fbcaf03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e2cf0a23c14e3e876406afcfdd6e07" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 33299 } } peers { permanent_uuid: "693a2a3014a34660ad9c23b17abceed7" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 46457 } } peers { permanent_uuid: "fb10411db4a0426fa59c5f405fbcaf03" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 34439 } }
I20260925 07:33:10.466502 25353 raft_consensus.cc:374] T 8fd7878e0398470aa866f1b32cd324ec P fb10411db4a0426fa59c5f405fbcaf03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:33:10.466524 25353 raft_consensus.cc:729] T 8fd7878e0398470aa866f1b32cd324ec P fb10411db4a0426fa59c5f405fbcaf03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb10411db4a0426fa59c5f405fbcaf03, State: Initialized, Role: FOLLOWER
I20260925 07:33:10.466650 25353 consensus_queue.cc:260] T 8fd7878e0398470aa866f1b32cd324ec P fb10411db4a0426fa59c5f405fbcaf03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e2cf0a23c14e3e876406afcfdd6e07" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 33299 } } peers { permanent_uuid: "693a2a3014a34660ad9c23b17abceed7" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 46457 } } peers { permanent_uuid: "fb10411db4a0426fa59c5f405fbcaf03" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 34439 } }
I20260925 07:33:10.466863 25353 ts_tablet_manager.cc:1428] T 8fd7878e0398470aa866f1b32cd324ec P fb10411db4a0426fa59c5f405fbcaf03: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 07:33:10.466930 25067 heartbeater.cc:503] Master 127.9.66.62:35239 was elected leader, sending a full tablet report...
W20260925 07:33:10.521019 25202 tablet.cc:2369] T 8fd7878e0398470aa866f1b32cd324ec P a3e2cf0a23c14e3e876406afcfdd6e07: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 07:33:10.620795 25068 tablet.cc:2369] T 8fd7878e0398470aa866f1b32cd324ec P fb10411db4a0426fa59c5f405fbcaf03: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 07:33:10.670110 25336 tablet.cc:2369] T 8fd7878e0398470aa866f1b32cd324ec P 693a2a3014a34660ad9c23b17abceed7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 07:33:10.709009 25355 raft_consensus.cc:482] T 8fd7878e0398470aa866f1b32cd324ec P a3e2cf0a23c14e3e876406afcfdd6e07 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 07:33:10.709131 25355 raft_consensus.cc:504] T 8fd7878e0398470aa866f1b32cd324ec P a3e2cf0a23c14e3e876406afcfdd6e07 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e2cf0a23c14e3e876406afcfdd6e07" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 33299 } } peers { permanent_uuid: "693a2a3014a34660ad9c23b17abceed7" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 46457 } } peers { permanent_uuid: "fb10411db4a0426fa59c5f405fbcaf03" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 34439 } }
I20260925 07:33:10.709434 25355 leader_election.cc:288] T 8fd7878e0398470aa866f1b32cd324ec P a3e2cf0a23c14e3e876406afcfdd6e07 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 693a2a3014a34660ad9c23b17abceed7 (127.9.66.3:46457), fb10411db4a0426fa59c5f405fbcaf03 (127.9.66.1:34439)
I20260925 07:33:10.713264 25203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:33:10.709665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.3:46457 (local address 127.9.66.2:50469)
0925 07:33:10.709727 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0925 07:33:10.709740 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 07:33:10.709815 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:33:10.709982 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:33:10.709995 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:33:10.709997 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:33:10.710080 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:33:10.710082 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:33:10.711024 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:33:10.711025 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:33:10.712217 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:33:10.712230 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:33:10.712992 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:33:10.712993 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:33:10.713061 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:33:10.713062 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 07:33:10.713121 (+ 59us) client_negotiation.cc:240] Negotiation successful
0925 07:33:10.713167 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":8}
I20260925 07:33:10.713271 25358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:33:10.709859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.1:34439 (local address 127.9.66.2:53137)
0925 07:33:10.710064 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0925 07:33:10.710074 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 07:33:10.710134 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:33:10.710268 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:33:10.710269 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:33:10.710270 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:33:10.710318 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:33:10.710319 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:33:10.711131 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:33:10.711132 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:33:10.712215 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:33:10.712230 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:33:10.712984 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:33:10.712984 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:33:10.713058 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:33:10.713062 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 07:33:10.713120 (+ 58us) client_negotiation.cc:240] Negotiation successful
0925 07:33:10.713166 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"mutex_wait_us":6,"thread_start_us":117,"threads_started":1}
I20260925 07:33:10.713311 25347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:33:10.709965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.2:53137 (local address 127.9.66.1:34439)
0925 07:33:10.710014 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 07:33:10.710016 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:33:10.710150 (+ 134us) server_negotiation.cc:407] Connection header received
0925 07:33:10.710182 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:33:10.710183 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:33:10.710194 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:33:10.710255 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:33:10.710376 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:33:10.711054 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:33:10.712324 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:33:10.712899 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:33:10.712968 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:33:10.712978 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 07:33:10.713178 (+ 200us) server_negotiation.cc:299] Negotiation successful
0925 07:33:10.713231 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 07:33:10.713311 25348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:33:10.709768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.2:50469 (local address 127.9.66.3:46457)
0925 07:33:10.709829 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 07:33:10.709831 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:33:10.709843 (+ 12us) server_negotiation.cc:407] Connection header received
0925 07:33:10.709877 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:33:10.709879 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:33:10.709891 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:33:10.709962 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:33:10.710175 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:33:10.710947 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:33:10.712324 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:33:10.712893 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:33:10.712968 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:33:10.712978 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 07:33:10.713178 (+ 200us) server_negotiation.cc:299] Negotiation successful
0925 07:33:10.713231 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 07:33:10.713698 25021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fd7878e0398470aa866f1b32cd324ec" candidate_uuid: "a3e2cf0a23c14e3e876406afcfdd6e07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb10411db4a0426fa59c5f405fbcaf03" is_pre_election: true
I20260925 07:33:10.713697 25289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fd7878e0398470aa866f1b32cd324ec" candidate_uuid: "a3e2cf0a23c14e3e876406afcfdd6e07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "693a2a3014a34660ad9c23b17abceed7" is_pre_election: true
I20260925 07:33:10.713852 25021 raft_consensus.cc:2454] T 8fd7878e0398470aa866f1b32cd324ec P fb10411db4a0426fa59c5f405fbcaf03 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3e2cf0a23c14e3e876406afcfdd6e07 in term 0.
I20260925 07:33:10.713852 25289 raft_consensus.cc:2454] T 8fd7878e0398470aa866f1b32cd324ec P 693a2a3014a34660ad9c23b17abceed7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3e2cf0a23c14e3e876406afcfdd6e07 in term 0.
I20260925 07:33:10.714063 25089 leader_election.cc:302] T 8fd7878e0398470aa866f1b32cd324ec P a3e2cf0a23c14e3e876406afcfdd6e07 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 693a2a3014a34660ad9c23b17abceed7, a3e2cf0a23c14e3e876406afcfdd6e07; no voters:
I20260925 07:33:10.714183 25355 raft_consensus.cc:2781] T 8fd7878e0398470aa866f1b32cd324ec P a3e2cf0a23c14e3e876406afcfdd6e07 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 07:33:10.714238 25355 raft_consensus.cc:482] T 8fd7878e0398470aa866f1b32cd324ec P a3e2cf0a23c14e3e876406afcfdd6e07 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 07:33:10.714267 25355 raft_consensus.cc:3037] T 8fd7878e0398470aa866f1b32cd324ec P a3e2cf0a23c14e3e876406afcfdd6e07 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:33:10.714926 25355 raft_consensus.cc:504] T 8fd7878e0398470aa866f1b32cd324ec P a3e2cf0a23c14e3e876406afcfdd6e07 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e2cf0a23c14e3e876406afcfdd6e07" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 33299 } } peers { permanent_uuid: "693a2a3014a34660ad9c23b17abceed7" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 46457 } } peers { permanent_uuid: "fb10411db4a0426fa59c5f405fbcaf03" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 34439 } }
I20260925 07:33:10.715052 25355 leader_election.cc:288] T 8fd7878e0398470aa866f1b32cd324ec P a3e2cf0a23c14e3e876406afcfdd6e07 [CANDIDATE]: Term 1 election: Requested vote from peers 693a2a3014a34660ad9c23b17abceed7 (127.9.66.3:46457), fb10411db4a0426fa59c5f405fbcaf03 (127.9.66.1:34439)
I20260925 07:33:10.715274 25289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fd7878e0398470aa866f1b32cd324ec" candidate_uuid: "a3e2cf0a23c14e3e876406afcfdd6e07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "693a2a3014a34660ad9c23b17abceed7"
I20260925 07:33:10.715288 25021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fd7878e0398470aa866f1b32cd324ec" candidate_uuid: "a3e2cf0a23c14e3e876406afcfdd6e07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb10411db4a0426fa59c5f405fbcaf03"
I20260925 07:33:10.715354 25289 raft_consensus.cc:3037] T 8fd7878e0398470aa866f1b32cd324ec P 693a2a3014a34660ad9c23b17abceed7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:33:10.715354 25021 raft_consensus.cc:3037] T 8fd7878e0398470aa866f1b32cd324ec P fb10411db4a0426fa59c5f405fbcaf03 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:33:10.716214 25289 raft_consensus.cc:2454] T 8fd7878e0398470aa866f1b32cd324ec P 693a2a3014a34660ad9c23b17abceed7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3e2cf0a23c14e3e876406afcfdd6e07 in term 1.
I20260925 07:33:10.716214 25021 raft_consensus.cc:2454] T 8fd7878e0398470aa866f1b32cd324ec P fb10411db4a0426fa59c5f405fbcaf03 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3e2cf0a23c14e3e876406afcfdd6e07 in term 1.
I20260925 07:33:10.716434 25089 leader_election.cc:302] T 8fd7878e0398470aa866f1b32cd324ec P a3e2cf0a23c14e3e876406afcfdd6e07 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 693a2a3014a34660ad9c23b17abceed7, a3e2cf0a23c14e3e876406afcfdd6e07; no voters:
I20260925 07:33:10.716517 25355 raft_consensus.cc:2781] T 8fd7878e0398470aa866f1b32cd324ec P a3e2cf0a23c14e3e876406afcfdd6e07 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:33:10.716648 25355 raft_consensus.cc:686] T 8fd7878e0398470aa866f1b32cd324ec P a3e2cf0a23c14e3e876406afcfdd6e07 [term 1 LEADER]: Becoming Leader. State: Replica: a3e2cf0a23c14e3e876406afcfdd6e07, State: Running, Role: LEADER
I20260925 07:33:10.716724 25355 consensus_queue.cc:237] T 8fd7878e0398470aa866f1b32cd324ec P a3e2cf0a23c14e3e876406afcfdd6e07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e2cf0a23c14e3e876406afcfdd6e07" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 33299 } } peers { permanent_uuid: "693a2a3014a34660ad9c23b17abceed7" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 46457 } } peers { permanent_uuid: "fb10411db4a0426fa59c5f405fbcaf03" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 34439 } }
I20260925 07:33:10.717640 24875 catalog_manager.cc:5495] T 8fd7878e0398470aa866f1b32cd324ec P a3e2cf0a23c14e3e876406afcfdd6e07 reported cstate change: term changed from 0 to 1, leader changed from <none> to a3e2cf0a23c14e3e876406afcfdd6e07 (127.9.66.2). New cstate: current_term: 1 leader_uuid: "a3e2cf0a23c14e3e876406afcfdd6e07" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e2cf0a23c14e3e876406afcfdd6e07" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 33299 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "693a2a3014a34660ad9c23b17abceed7" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 46457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb10411db4a0426fa59c5f405fbcaf03" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 34439 } health_report { overall_health: UNKNOWN } } }
I20260925 07:33:10.742112 25344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:33:10.739546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38076 (local address 127.9.66.2:33299)
0925 07:33:10.739651 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0925 07:33:10.739653 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:33:10.739663 (+ 10us) server_negotiation.cc:407] Connection header received
0925 07:33:10.739697 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:33:10.739697 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:33:10.739717 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:33:10.739794 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 07:33:10.739887 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:33:10.740451 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:33:10.741149 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:33:10.741349 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:33:10.741441 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:33:10.741756 (+ 315us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 07:33:10.741861 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 07:33:10.741952 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 07:33:10.741974 (+ 22us) server_negotiation.cc:299] Negotiation successful
0925 07:33:10.741996 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260925 07:33:10.744014 25289 raft_consensus.cc:1261] T 8fd7878e0398470aa866f1b32cd324ec P 693a2a3014a34660ad9c23b17abceed7 [term 1 FOLLOWER]: Refusing update from remote peer a3e2cf0a23c14e3e876406afcfdd6e07: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:33:10.744014 25021 raft_consensus.cc:1261] T 8fd7878e0398470aa866f1b32cd324ec P fb10411db4a0426fa59c5f405fbcaf03 [term 1 FOLLOWER]: Refusing update from remote peer a3e2cf0a23c14e3e876406afcfdd6e07: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:33:10.744390 25355 consensus_queue.cc:1035] T 8fd7878e0398470aa866f1b32cd324ec P a3e2cf0a23c14e3e876406afcfdd6e07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "693a2a3014a34660ad9c23b17abceed7" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 46457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:33:10.744527 25359 consensus_queue.cc:1035] T 8fd7878e0398470aa866f1b32cd324ec P a3e2cf0a23c14e3e876406afcfdd6e07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb10411db4a0426fa59c5f405fbcaf03" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 34439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:33:10.746977 25363 mvcc.cc:205] Tried to move back new op lower bound from 7333157235684474880 to 7333157235576143872. Current Snapshot: MvccSnapshot[applied={T|T < 7333157235684474880}]
I20260925 07:33:10.747512 25364 mvcc.cc:205] Tried to move back new op lower bound from 7333157235684474880 to 7333157235576143872. Current Snapshot: MvccSnapshot[applied={T|T < 7333157235684474880}]
I20260925 07:33:10.749053 24875 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46308:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 07:33:10.749146 24875 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46308:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 07:33:10.749830 24875 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 10c9b33b17904def90c291c5a7091209: Sending DeleteTablet for 3 replicas of tablet 8fd7878e0398470aa866f1b32cd324ec
I20260925 07:33:10.750183 25135 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fd7878e0398470aa866f1b32cd324ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:33:10 UTC) from {username='kudu', principal='kudu/127.9.66.62@KRBTEST.COM'} at 127.0.0.1:38064
I20260925 07:33:10.750283 25269 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fd7878e0398470aa866f1b32cd324ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:33:10 UTC) from {username='kudu', principal='kudu/127.9.66.62@KRBTEST.COM'} at 127.0.0.1:35266
I20260925 07:33:10.750316 25001 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fd7878e0398470aa866f1b32cd324ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:33:10 UTC) from {username='kudu', principal='kudu/127.9.66.62@KRBTEST.COM'} at 127.0.0.1:35504
I20260925 07:33:10.750505 25374 tablet_replica.cc:330] T 8fd7878e0398470aa866f1b32cd324ec P fb10411db4a0426fa59c5f405fbcaf03: stopping tablet replica
I20260925 07:33:10.750531 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 24938
I20260925 07:33:10.750595 25374 raft_consensus.cc:2229] T 8fd7878e0398470aa866f1b32cd324ec P fb10411db4a0426fa59c5f405fbcaf03 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 07:33:10.750696 25374 raft_consensus.cc:2258] T 8fd7878e0398470aa866f1b32cd324ec P fb10411db4a0426fa59c5f405fbcaf03 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:33:10.753638 25372 tablet_replica.cc:330] T 8fd7878e0398470aa866f1b32cd324ec P a3e2cf0a23c14e3e876406afcfdd6e07: stopping tablet replica
I20260925 07:33:10.753737 25372 raft_consensus.cc:2229] T 8fd7878e0398470aa866f1b32cd324ec P a3e2cf0a23c14e3e876406afcfdd6e07 [term 1 LEADER]: Raft consensus shutting down.
I20260925 07:33:10.753847 25372 raft_consensus.cc:2258] T 8fd7878e0398470aa866f1b32cd324ec P a3e2cf0a23c14e3e876406afcfdd6e07 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:33:10.753984 25373 tablet_replica.cc:330] T 8fd7878e0398470aa866f1b32cd324ec P 693a2a3014a34660ad9c23b17abceed7: stopping tablet replica
I20260925 07:33:10.754076 25373 raft_consensus.cc:2229] T 8fd7878e0398470aa866f1b32cd324ec P 693a2a3014a34660ad9c23b17abceed7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 07:33:10.754169 25373 raft_consensus.cc:2258] T 8fd7878e0398470aa866f1b32cd324ec P 693a2a3014a34660ad9c23b17abceed7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:33:10.754575 25373 ts_tablet_manager.cc:1905] T 8fd7878e0398470aa866f1b32cd324ec P 693a2a3014a34660ad9c23b17abceed7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 07:33:10.756201 25373 ts_tablet_manager.cc:1918] T 8fd7878e0398470aa866f1b32cd324ec P 693a2a3014a34660ad9c23b17abceed7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 07:33:10.756268 25373 log.cc:1196] T 8fd7878e0398470aa866f1b32cd324ec P 693a2a3014a34660ad9c23b17abceed7: Deleting WAL directory at /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-2/wal/wals/8fd7878e0398470aa866f1b32cd324ec
W20260925 07:33:10.756656 24861 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.66.1:34439 (error 108)
I20260925 07:33:10.756544 25373 ts_tablet_manager.cc:1939] T 8fd7878e0398470aa866f1b32cd324ec P 693a2a3014a34660ad9c23b17abceed7: Deleting consensus metadata
I20260925 07:33:10.756799 25372 ts_tablet_manager.cc:1905] T 8fd7878e0398470aa866f1b32cd324ec P a3e2cf0a23c14e3e876406afcfdd6e07: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 07:33:10.757071 24861 catalog_manager.cc:4841] TS 693a2a3014a34660ad9c23b17abceed7 (127.9.66.3:46457): tablet 8fd7878e0398470aa866f1b32cd324ec (table test-table [id=b9b78c1e10a94a52bdc4bc3e3baf439f]) successfully deleted
I20260925 07:33:10.757274 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 25072
I20260925 07:33:10.758128 25346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:33:10.757366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.1:34439 (local address 127.0.0.1:35518)
0925 07:33:10.758066 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0925 07:33:10.758102 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.1:34439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":681}
W20260925 07:33:10.758247 24861 catalog_manager.cc:4584] TS fb10411db4a0426fa59c5f405fbcaf03 (127.9.66.1:34439): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8fd7878e0398470aa866f1b32cd324ec: Network error: Client connection negotiation failed: client connection to 127.9.66.1:34439: connect: Connection refused (error 111)
I20260925 07:33:10.763525 25346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:33:10.763428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.2:33299 (local address 127.0.0.1:38080)
0925 07:33:10.763470 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0925 07:33:10.763502 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.2:33299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260925 07:33:10.763533 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 25206
I20260925 07:33:10.768061 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 24844
I20260925 07:33:30.773350 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.66.62:35239
--webserver_interface=127.9.66.62
--webserver_port=46119
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab
--principal=kudu/127.9.66.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:43203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.66.62:35239
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:33:30.852308 25376 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:33:30.852500 25376 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:33:30.852521 25376 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:33:30.853688 25376 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 07:33:30.853724 25376 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 07:33:30.853737 25376 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 07:33:30.853749 25376 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:33:30.853806 25376 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:33:30.853822 25376 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:33:30.853847 25376 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:33:30.853875 25376 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:33:30.855305 25376 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:43203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.9.66.62:35239
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab
--principal=kudu/127.9.66.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.66.62:35239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.66.62
--webserver_port=46119
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25376
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:33:30.855557 25376 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:33:30.855784 25376 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:33:30.864138 25385 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:33:30.864144 25382 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:33:30.864223 25383 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:33:30.864399 25376 server_base.cc:1034] running on GCE node
I20260925 07:33:30.864630 25376 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:33:30.864904 25376 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:33:30.866047 25376 hybrid_clock.cc:648] HybridClock initialized: now 1790321610866029 us; error 35 us; skew 500 ppm
Sep 25 07:33:30 dist-test-slave-57fn krb5kdc[24831](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321610, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:33:30.868598 25376 init.cc:377] Logged in from keytab as kudu/127.9.66.62@KRBTEST.COM (short username kudu)
I20260925 07:33:30.869063 25376 webserver.cc:460] Webserver started at http://127.9.66.62:46119/ using document root <none> and password file <none>
I20260925 07:33:30.869290 25376 fs_manager.cc:359] Metadata directory not provided
I20260925 07:33:30.869362 25376 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:33:30.870697 25376 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 07:33:30.871454 25392 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:33:30.871641 25376 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 07:33:30.871721 25376 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/master-0/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/master-0/wal
uuid: "10c9b33b17904def90c291c5a7091209"
format_stamp: "Formatted at 2026-09-25 07:33:09 on dist-test-slave-57fn"
server_key: "41bae02c09abafa675bad2414d6b6471"
server_key_iv: "7b491795d817902dbd50efa44281d65e"
server_key_version: "encryptionkey@0"
I20260925 07:33:30.872107 25376 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:33:30.880939 25376 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:33:30.882614 25376 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:33:30.882715 25376 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:33:30.886364 25376 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.62:35239
I20260925 07:33:30.886413 25444 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.62:35239 every 8 connection(s)
I20260925 07:33:30.886749 25376 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
I20260925 07:33:30.888118 25445 sys_catalog.cc:263] Verifying existing consensus state
I20260925 07:33:30.888319 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 25376
I20260925 07:33:30.888576 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.66.1:34439
--local_ip_for_outbound_sockets=127.9.66.1
--tserver_master_addrs=127.9.66.62:35239
--webserver_port=32927
--webserver_interface=127.9.66.1
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.1.keytab
--principal=kudu/127.9.66.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:43203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 07:33:30.888885 25445 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10c9b33b17904def90c291c5a7091209: Bootstrap starting.
I20260925 07:33:30.897526 25445 log.cc:824] T 00000000000000000000000000000000 P 10c9b33b17904def90c291c5a7091209: Log is configured to *not* fsync() on all Append() calls
I20260925 07:33:30.899370 25445 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10c9b33b17904def90c291c5a7091209: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 07:33:30.899653 25445 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10c9b33b17904def90c291c5a7091209: Bootstrap complete.
I20260925 07:33:30.902032 25445 raft_consensus.cc:348] T 00000000000000000000000000000000 P 10c9b33b17904def90c291c5a7091209 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10c9b33b17904def90c291c5a7091209" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 35239 } }
I20260925 07:33:30.902271 25445 raft_consensus.cc:729] T 00000000000000000000000000000000 P 10c9b33b17904def90c291c5a7091209 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10c9b33b17904def90c291c5a7091209, State: Initialized, Role: FOLLOWER
I20260925 07:33:30.902479 25445 consensus_queue.cc:260] T 00000000000000000000000000000000 P 10c9b33b17904def90c291c5a7091209 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10c9b33b17904def90c291c5a7091209" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 35239 } }
I20260925 07:33:30.902560 25445 raft_consensus.cc:388] T 00000000000000000000000000000000 P 10c9b33b17904def90c291c5a7091209 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 07:33:30.902616 25445 raft_consensus.cc:482] T 00000000000000000000000000000000 P 10c9b33b17904def90c291c5a7091209 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 07:33:30.902667 25445 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 10c9b33b17904def90c291c5a7091209 [term 1 FOLLOWER]: Advancing to term 2
I20260925 07:33:30.903638 25445 raft_consensus.cc:504] T 00000000000000000000000000000000 P 10c9b33b17904def90c291c5a7091209 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10c9b33b17904def90c291c5a7091209" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 35239 } }
I20260925 07:33:30.903766 25445 leader_election.cc:302] T 00000000000000000000000000000000 P 10c9b33b17904def90c291c5a7091209 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 10c9b33b17904def90c291c5a7091209; no voters:
I20260925 07:33:30.903980 25445 leader_election.cc:288] T 00000000000000000000000000000000 P 10c9b33b17904def90c291c5a7091209 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 07:33:30.904011 25449 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 10c9b33b17904def90c291c5a7091209 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 07:33:30.904145 25449 raft_consensus.cc:686] T 00000000000000000000000000000000 P 10c9b33b17904def90c291c5a7091209 [term 2 LEADER]: Becoming Leader. State: Replica: 10c9b33b17904def90c291c5a7091209, State: Running, Role: LEADER
I20260925 07:33:30.904250 25445 sys_catalog.cc:564] T 00000000000000000000000000000000 P 10c9b33b17904def90c291c5a7091209 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:33:30.904320 25449 consensus_queue.cc:237] T 00000000000000000000000000000000 P 10c9b33b17904def90c291c5a7091209 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10c9b33b17904def90c291c5a7091209" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 35239 } }
I20260925 07:33:30.904678 25450 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10c9b33b17904def90c291c5a7091209 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "10c9b33b17904def90c291c5a7091209" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10c9b33b17904def90c291c5a7091209" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 35239 } } }
I20260925 07:33:30.904770 25450 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10c9b33b17904def90c291c5a7091209 [sys.catalog]: This master's current role is: LEADER
I20260925 07:33:30.905077 25451 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10c9b33b17904def90c291c5a7091209 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 10c9b33b17904def90c291c5a7091209. Latest consensus state: current_term: 2 leader_uuid: "10c9b33b17904def90c291c5a7091209" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10c9b33b17904def90c291c5a7091209" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 35239 } } }
I20260925 07:33:30.905139 25451 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10c9b33b17904def90c291c5a7091209 [sys.catalog]: This master's current role is: LEADER
I20260925 07:33:30.906049 25465 catalog_manager.cc:1259] Loaded cluster ID: dd1c0386aa0144abaed6dc16b9d4fffc
I20260925 07:33:30.906109 25465 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 10c9b33b17904def90c291c5a7091209: loading cluster ID for follower catalog manager: success
I20260925 07:33:30.906978 25465 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 10c9b33b17904def90c291c5a7091209: acquiring CA information for follower catalog manager: success
I20260925 07:33:30.907347 25465 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 10c9b33b17904def90c291c5a7091209: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 07:33:30.907439 25456 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 07:33:30.907824 25456 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 07:33:30.907902 25456 catalog_manager.cc:1259] Loaded cluster ID: dd1c0386aa0144abaed6dc16b9d4fffc
I20260925 07:33:30.907933 25456 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 07:33:30.908088 25456 catalog_manager.cc:1504] Loading token signing keys...
I20260925 07:33:30.908166 25456 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 10c9b33b17904def90c291c5a7091209: Loaded TSK: 0
I20260925 07:33:30.924405 25456 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 10c9b33b17904def90c291c5a7091209: Generated new TSK 1
I20260925 07:33:30.924507 25456 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 07:33:30.973487 25447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:33:30.973724 25447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:33:30.973747 25447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:33:30.974960 25447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:33:30.975008 25447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:33:30.975031 25447 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.66.1
I20260925 07:33:30.976452 25447 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:43203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.1.keytab
--principal=kudu/127.9.66.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.66.1:34439
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.66.1
--webserver_port=32927
--tserver_master_addrs=127.9.66.62:35239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25447
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.66.1
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:33:30.976677 25447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:33:30.976908 25447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:33:30.981717 25472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:33:30.981736 25475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:33:30.981703 25473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:33:30.981979 25447 server_base.cc:1034] running on GCE node
I20260925 07:33:30.982172 25447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:33:30.982417 25447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:33:30.983589 25447 hybrid_clock.cc:648] HybridClock initialized: now 1790321610983579 us; error 31 us; skew 500 ppm
Sep 25 07:33:30 dist-test-slave-57fn krb5kdc[24831](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321610, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:33:30.985931 25447 init.cc:377] Logged in from keytab as kudu/127.9.66.1@KRBTEST.COM (short username kudu)
I20260925 07:33:30.986385 25447 webserver.cc:460] Webserver started at http://127.9.66.1:32927/ using document root <none> and password file <none>
I20260925 07:33:30.986609 25447 fs_manager.cc:359] Metadata directory not provided
I20260925 07:33:30.986686 25447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:33:30.987995 25447 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:33:30.988688 25482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:33:30.988858 25447 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 07:33:30.988930 25447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-0/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-0/wal
uuid: "fb10411db4a0426fa59c5f405fbcaf03"
format_stamp: "Formatted at 2026-09-25 07:33:10 on dist-test-slave-57fn"
server_key: "896b5afaa9608339fc8459e883be320b"
server_key_iv: "5adf2d1d9de966513d2a996252efe614"
server_key_version: "encryptionkey@0"
I20260925 07:33:30.989234 25447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:33:31.000898 25447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:33:31.003090 25447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:33:31.003225 25447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:33:31.003448 25447 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:33:31.003885 25489 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 07:33:31.005757 25447 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 07:33:31.005813 25447 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260925 07:33:31.005856 25447 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 07:33:31.006430 25447 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 07:33:31.006484 25447 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260925 07:33:31.006557 25489 tablet_bootstrap.cc:492] T 8fd7878e0398470aa866f1b32cd324ec P fb10411db4a0426fa59c5f405fbcaf03: Bootstrap starting.
I20260925 07:33:31.013393 25447 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.1:34439
I20260925 07:33:31.013585 25596 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.1:34439 every 8 connection(s)
I20260925 07:33:31.013834 25447 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-0/data/info.pb
I20260925 07:33:31.017048 25489 log.cc:824] T 8fd7878e0398470aa866f1b32cd324ec P fb10411db4a0426fa59c5f405fbcaf03: Log is configured to *not* fsync() on all Append() calls
I20260925 07:33:31.019879 25489 tablet_bootstrap.cc:492] T 8fd7878e0398470aa866f1b32cd324ec P fb10411db4a0426fa59c5f405fbcaf03: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 07:33:31.020186 25489 tablet_bootstrap.cc:492] T 8fd7878e0398470aa866f1b32cd324ec P fb10411db4a0426fa59c5f405fbcaf03: Bootstrap complete.
I20260925 07:33:31.021131 25489 ts_tablet_manager.cc:1397] T 8fd7878e0398470aa866f1b32cd324ec P fb10411db4a0426fa59c5f405fbcaf03: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.010s
I20260925 07:33:31.022337 25489 raft_consensus.cc:348] T 8fd7878e0398470aa866f1b32cd324ec P fb10411db4a0426fa59c5f405fbcaf03 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e2cf0a23c14e3e876406afcfdd6e07" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 33299 } } peers { permanent_uuid: "693a2a3014a34660ad9c23b17abceed7" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 46457 } } peers { permanent_uuid: "fb10411db4a0426fa59c5f405fbcaf03" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 34439 } }
I20260925 07:33:31.022671 25489 raft_consensus.cc:729] T 8fd7878e0398470aa866f1b32cd324ec P fb10411db4a0426fa59c5f405fbcaf03 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb10411db4a0426fa59c5f405fbcaf03, State: Initialized, Role: FOLLOWER
I20260925 07:33:31.022830 25489 consensus_queue.cc:260] T 8fd7878e0398470aa866f1b32cd324ec P fb10411db4a0426fa59c5f405fbcaf03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e2cf0a23c14e3e876406afcfdd6e07" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 33299 } } peers { permanent_uuid: "693a2a3014a34660ad9c23b17abceed7" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 46457 } } peers { permanent_uuid: "fb10411db4a0426fa59c5f405fbcaf03" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 34439 } }
I20260925 07:33:31.023106 25489 ts_tablet_manager.cc:1428] T 8fd7878e0398470aa866f1b32cd324ec P fb10411db4a0426fa59c5f405fbcaf03: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 07:33:31.023674 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 25447
I20260925 07:33:31.023967 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.66.2:33299
--local_ip_for_outbound_sockets=127.9.66.2
--tserver_master_addrs=127.9.66.62:35239
--webserver_port=34779
--webserver_interface=127.9.66.2
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.2.keytab
--principal=kudu/127.9.66.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:43203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 07:33:31 dist-test-slave-57fn krb5kdc[24831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321610, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.1@KRBTEST.COM for kudu/127.9.66.62@KRBTEST.COM
I20260925 07:33:31.030911 25600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:33:31.014529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.1:58377 (local address 127.9.66.62:35239)
0925 07:33:31.019342 (+ 4813us) server_negotiation.cc:206] Beginning negotiation
0925 07:33:31.019350 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 07:33:31.019383 (+ 33us) server_negotiation.cc:407] Connection header received
0925 07:33:31.019507 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:33:31.019513 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:33:31.019566 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:33:31.019705 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:33:31.020501 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:33:31.021816 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:33:31.023208 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:33:31.023776 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:33:31.023877 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:33:31.026775 (+ 2898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:33:31.026780 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:33:31.026780 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:33:31.026793 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:33:31.028826 (+ 2033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:33:31.029351 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:33:31.029354 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:33:31.029354 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:33:31.029468 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:33:31.029852 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:33:31.029853 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:33:31.029854 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:33:31.030131 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:33:31.030217 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 07:33:31.030579 (+ 362us) server_negotiation.cc:299] Negotiation successful
0925 07:33:31.030717 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4764,"thread_start_us":71,"threads_started":1}
I20260925 07:33:31.030932 25599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:33:31.014472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:35239 (local address 127.9.66.1:58377)
0925 07:33:31.017346 (+ 2874us) negotiation.cc:107] Waiting for socket to connect
0925 07:33:31.017388 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 07:33:31.019020 (+ 1632us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:33:31.019766 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:33:31.019770 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:33:31.020134 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:33:31.020386 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:33:31.020391 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:33:31.021915 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:33:31.021917 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:33:31.023110 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:33:31.023112 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:33:31.023861 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:33:31.023863 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:33:31.024020 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:33:31.024732 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:33:31.024742 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:33:31.026643 (+ 1901us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:33:31.028913 (+ 2270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:33:31.028920 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:33:31.028922 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:33:31.029256 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:33:31.029626 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:33:31.029627 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:33:31.029627 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:33:31.029771 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:33:31.030213 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:33:31.030219 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:33:31.030384 (+ 165us) client_negotiation.cc:769] Sending connection context
0925 07:33:31.030507 (+ 123us) client_negotiation.cc:240] Negotiation successful
0925 07:33:31.030684 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2835,"thread_start_us":100,"threads_started":1}
I20260925 07:33:31.032404 25597 heartbeater.cc:347] Connected to a master server at 127.9.66.62:35239
I20260925 07:33:31.032505 25597 heartbeater.cc:464] Registering TS with master...
I20260925 07:33:31.032718 25597 heartbeater.cc:511] Master 127.9.66.62:35239 requested a full tablet report, sending...
I20260925 07:33:31.033492 25409 ts_manager.cc:194] Registered new tserver with Master: fb10411db4a0426fa59c5f405fbcaf03 (127.9.66.1:34439)
I20260925 07:33:31.034662 25409 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.66.1@KRBTEST.COM'} at 127.9.66.1:58377
I20260925 07:33:31.035567 25597 heartbeater.cc:503] Master 127.9.66.62:35239 was elected leader, sending a full tablet report...
I20260925 07:33:31.040089 25605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:33:31.034903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59004 (local address 127.9.66.1:34439)
0925 07:33:31.035759 (+ 856us) server_negotiation.cc:206] Beginning negotiation
0925 07:33:31.035763 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 07:33:31.036067 (+ 304us) server_negotiation.cc:407] Connection header received
0925 07:33:31.036152 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:33:31.036155 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:33:31.036169 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:33:31.036318 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:33:31.036479 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:33:31.037647 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:33:31.039169 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:33:31.039716 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:33:31.039788 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:33:31.039832 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0925 07:33:31.039996 (+ 164us) server_negotiation.cc:299] Negotiation successful
0925 07:33:31.040031 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":831,"thread_start_us":78,"threads_started":1}
I20260925 07:33:31.040092 25604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:33:31.034089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.1:34439 (local address 127.0.0.1:59004)
0925 07:33:31.035080 (+ 991us) negotiation.cc:107] Waiting for socket to connect
0925 07:33:31.035930 (+ 850us) client_negotiation.cc:174] Beginning negotiation
0925 07:33:31.035992 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:33:31.036342 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:33:31.036345 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:33:31.036347 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:33:31.036414 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:33:31.036415 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:33:31.037725 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:33:31.037726 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:33:31.038968 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:33:31.038971 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:33:31.039798 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:33:31.039799 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:33:31.039885 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:33:31.039898 (+ 13us) client_negotiation.cc:769] Sending connection context
0925 07:33:31.039968 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 07:33:31.040014 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":943,"thread_start_us":72,"threads_started":1}
I20260925 07:33:31.040681 25529 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fd7878e0398470aa866f1b32cd324ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:33:10 UTC) from {username='kudu', principal='kudu/127.9.66.62@KRBTEST.COM'} at 127.0.0.1:59004
I20260925 07:33:31.041174 25606 tablet_replica.cc:330] stopping tablet replica
I20260925 07:33:31.041299 25606 raft_consensus.cc:2229] T 8fd7878e0398470aa866f1b32cd324ec P fb10411db4a0426fa59c5f405fbcaf03 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 07:33:31.041394 25606 raft_consensus.cc:2258] T 8fd7878e0398470aa866f1b32cd324ec P fb10411db4a0426fa59c5f405fbcaf03 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:33:31.041807 25606 ts_tablet_manager.cc:1905] T 8fd7878e0398470aa866f1b32cd324ec P fb10411db4a0426fa59c5f405fbcaf03: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 07:33:31.044260 25606 ts_tablet_manager.cc:1918] T 8fd7878e0398470aa866f1b32cd324ec P fb10411db4a0426fa59c5f405fbcaf03: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 07:33:31.044333 25606 log.cc:1196] T 8fd7878e0398470aa866f1b32cd324ec P fb10411db4a0426fa59c5f405fbcaf03: Deleting WAL directory at /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-0/wal/wals/8fd7878e0398470aa866f1b32cd324ec
I20260925 07:33:31.044553 25606 ts_tablet_manager.cc:1939] T 8fd7878e0398470aa866f1b32cd324ec P fb10411db4a0426fa59c5f405fbcaf03: Deleting consensus metadata
I20260925 07:33:31.045085 25394 catalog_manager.cc:4841] TS fb10411db4a0426fa59c5f405fbcaf03 (127.9.66.1:34439): tablet 8fd7878e0398470aa866f1b32cd324ec (table test-table [id=b9b78c1e10a94a52bdc4bc3e3baf439f]) successfully deleted
W20260925 07:33:31.106571 25603 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:33:31.106757 25603 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:33:31.106781 25603 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:33:31.107846 25603 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:33:31.107879 25603 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:33:31.107904 25603 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.66.2
I20260925 07:33:31.109153 25603 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:43203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.2.keytab
--principal=kudu/127.9.66.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.66.2:33299
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.66.2
--webserver_port=34779
--tserver_master_addrs=127.9.66.62:35239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25603
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.66.2
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:33:31.109329 25603 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:33:31.109473 25603 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:33:31.114171 25614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:33:31.114157 25612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:33:31.114197 25603 server_base.cc:1034] running on GCE node
W20260925 07:33:31.114180 25611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:33:31.114518 25603 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:33:31.114792 25603 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:33:31.115954 25603 hybrid_clock.cc:648] HybridClock initialized: now 1790321611115935 us; error 35 us; skew 500 ppm
Sep 25 07:33:31 dist-test-slave-57fn krb5kdc[24831](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321611, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:33:31.118422 25603 init.cc:377] Logged in from keytab as kudu/127.9.66.2@KRBTEST.COM (short username kudu)
I20260925 07:33:31.118803 25603 webserver.cc:460] Webserver started at http://127.9.66.2:34779/ using document root <none> and password file <none>
I20260925 07:33:31.119004 25603 fs_manager.cc:359] Metadata directory not provided
I20260925 07:33:31.119122 25603 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:33:31.120524 25603 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 07:33:31.121085 25621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:33:31.121232 25603 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 07:33:31.121312 25603 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-1/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-1/wal
uuid: "a3e2cf0a23c14e3e876406afcfdd6e07"
format_stamp: "Formatted at 2026-09-25 07:33:10 on dist-test-slave-57fn"
server_key: "249ac6722ef820f6a1335e766d83a839"
server_key_iv: "42dc75f8fcd51bab6b2ab4736611a7c7"
server_key_version: "encryptionkey@0"
I20260925 07:33:31.121630 25603 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:33:31.131616 25603 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:33:31.133370 25603 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:33:31.133569 25603 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:33:31.133786 25603 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:33:31.134176 25628 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260925 07:33:31.135483 25628 ts_tablet_manager.cc:1852] T 8fd7878e0398470aa866f1b32cd324ec P a3e2cf0a23c14e3e876406afcfdd6e07: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260925 07:33:31.135743 25628 ts_tablet_manager.cc:1905] T 8fd7878e0398470aa866f1b32cd324ec P a3e2cf0a23c14e3e876406afcfdd6e07: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 07:33:31.137805 25628 ts_tablet_manager.cc:1918] T 8fd7878e0398470aa866f1b32cd324ec P a3e2cf0a23c14e3e876406afcfdd6e07: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 07:33:31.137872 25628 log.cc:1196] T 8fd7878e0398470aa866f1b32cd324ec P a3e2cf0a23c14e3e876406afcfdd6e07: Deleting WAL directory at /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-1/wal/wals/8fd7878e0398470aa866f1b32cd324ec
I20260925 07:33:31.138149 25628 ts_tablet_manager.cc:1939] T 8fd7878e0398470aa866f1b32cd324ec P a3e2cf0a23c14e3e876406afcfdd6e07: Deleting consensus metadata
I20260925 07:33:31.138345 25603 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260925 07:33:31.138398 25603 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260925 07:33:31.138437 25603 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:33:31.138464 25603 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:33:31.144309 25603 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.2:33299
I20260925 07:33:31.144402 25735 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.2:33299 every 8 connection(s)
I20260925 07:33:31.144647 25603 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-1/data/info.pb
I20260925 07:33:31.150273 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 25603
Sep 25 07:33:31 dist-test-slave-57fn krb5kdc[24831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321611, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.2@KRBTEST.COM for kudu/127.9.66.62@KRBTEST.COM
I20260925 07:33:31.150535 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.66.3:46457
--local_ip_for_outbound_sockets=127.9.66.3
--tserver_master_addrs=127.9.66.62:35239
--webserver_port=40087
--webserver_interface=127.9.66.3
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.3.keytab
--principal=kudu/127.9.66.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:43203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 07:33:31.153982 25738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:33:31.145342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:35239 (local address 127.9.66.2:36467)
0925 07:33:31.145593 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0925 07:33:31.145630 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 07:33:31.146068 (+ 438us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:33:31.146257 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:33:31.146260 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:33:31.146485 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:33:31.146655 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:33:31.146658 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:33:31.147701 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:33:31.147703 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:33:31.148547 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:33:31.148549 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:33:31.148939 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:33:31.148940 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:33:31.149071 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:33:31.149498 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:33:31.149544 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:33:31.150909 (+ 1365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:33:31.152723 (+ 1814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:33:31.152728 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:33:31.152730 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:33:31.152931 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:33:31.153127 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:33:31.153128 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:33:31.153128 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:33:31.153215 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:33:31.153498 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:33:31.153503 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:33:31.153660 (+ 157us) client_negotiation.cc:769] Sending connection context
0925 07:33:31.153734 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 07:33:31.153832 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":132,"threads_started":1}
I20260925 07:33:31.154007 25600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:33:31.145437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.2:36467 (local address 127.9.66.62:35239)
0925 07:33:31.145542 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0925 07:33:31.145544 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:33:31.145709 (+ 165us) server_negotiation.cc:407] Connection header received
0925 07:33:31.146150 (+ 441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:33:31.146152 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:33:31.146162 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:33:31.146212 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:33:31.146796 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:33:31.147625 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:33:31.148638 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:33:31.148857 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:33:31.148934 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:33:31.151004 (+ 2070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:33:31.151005 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:33:31.151006 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:33:31.151012 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:33:31.152641 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:33:31.152999 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:33:31.153001 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:33:31.153001 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:33:31.153061 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:33:31.153276 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:33:31.153277 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:33:31.153278 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:33:31.153427 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:33:31.153561 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0925 07:33:31.153778 (+ 217us) server_negotiation.cc:299] Negotiation successful
0925 07:33:31.153864 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 07:33:31.154440 25736 heartbeater.cc:347] Connected to a master server at 127.9.66.62:35239
I20260925 07:33:31.154534 25736 heartbeater.cc:464] Registering TS with master...
I20260925 07:33:31.154704 25736 heartbeater.cc:511] Master 127.9.66.62:35239 requested a full tablet report, sending...
I20260925 07:33:31.155017 25409 ts_manager.cc:194] Registered new tserver with Master: a3e2cf0a23c14e3e876406afcfdd6e07 (127.9.66.2:33299)
I20260925 07:33:31.155360 25409 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.66.2@KRBTEST.COM'} at 127.9.66.2:36467
W20260925 07:33:31.228562 25739 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:33:31.228787 25739 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:33:31.228822 25739 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:33:31.229996 25739 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:33:31.230055 25739 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:33:31.230099 25739 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.66.3
I20260925 07:33:31.231446 25739 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:43203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.3.keytab
--principal=kudu/127.9.66.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.66.3:46457
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.66.3
--webserver_port=40087
--tserver_master_addrs=127.9.66.62:35239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25739
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.66.3
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:33:31.231686 25739 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:33:31.231894 25739 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 07:33:31.236730 25739 server_base.cc:1034] running on GCE node
W20260925 07:33:31.236727 25745 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:33:31.236792 25744 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:33:31.236864 25747 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:33:31.237080 25739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:33:31.237337 25739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:33:31.238523 25739 hybrid_clock.cc:648] HybridClock initialized: now 1790321611238508 us; error 24 us; skew 500 ppm
Sep 25 07:33:31 dist-test-slave-57fn krb5kdc[24831](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321611, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:33:31.241035 25739 init.cc:377] Logged in from keytab as kudu/127.9.66.3@KRBTEST.COM (short username kudu)
I20260925 07:33:31.241464 25739 webserver.cc:460] Webserver started at http://127.9.66.3:40087/ using document root <none> and password file <none>
I20260925 07:33:31.241689 25739 fs_manager.cc:359] Metadata directory not provided
I20260925 07:33:31.241778 25739 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:33:31.243023 25739 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:33:31.243564 25754 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:33:31.243731 25739 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 07:33:31.243811 25739 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-2/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-2/wal
uuid: "693a2a3014a34660ad9c23b17abceed7"
format_stamp: "Formatted at 2026-09-25 07:33:10 on dist-test-slave-57fn"
server_key: "40e264dce66644c9faa1b5b316241f02"
server_key_iv: "d0fb262afd73437e77767ec549298845"
server_key_version: "encryptionkey@0"
I20260925 07:33:31.244138 25739 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:33:31.265424 25739 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:33:31.267719 25739 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:33:31.267880 25739 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:33:31.268110 25739 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:33:31.268420 25739 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:33:31.268472 25739 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:33:31.268510 25739 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:33:31.268537 25739 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:33:31.274329 25739 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.3:46457
I20260925 07:33:31.274366 25867 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.3:46457 every 8 connection(s)
I20260925 07:33:31.274734 25739 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-2/data/info.pb
I20260925 07:33:31.275906 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 25739
Sep 25 07:33:31 dist-test-slave-57fn krb5kdc[24831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321611, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.3@KRBTEST.COM for kudu/127.9.66.62@KRBTEST.COM
I20260925 07:33:31.282824 25871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:33:31.276242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54086 (local address 127.9.66.62:35239)
0925 07:33:31.276328 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0925 07:33:31.276329 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 07:33:31.276490 (+ 161us) server_negotiation.cc:407] Connection header received
0925 07:33:31.276536 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:33:31.276537 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:33:31.276551 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:33:31.276621 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:33:31.277129 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:33:31.277702 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:33:31.278467 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:33:31.278743 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:33:31.278816 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:33:31.279999 (+ 1183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:33:31.280000 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:33:31.280001 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:33:31.280007 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:33:31.281587 (+ 1580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:33:31.281975 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:33:31.281976 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:33:31.281976 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:33:31.282035 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:33:31.282266 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:33:31.282267 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:33:31.282267 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:33:31.282423 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:33:31.282597 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0925 07:33:31.282638 (+ 41us) server_negotiation.cc:299] Negotiation successful
0925 07:33:31.282706 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69,"thread_start_us":44,"threads_started":1}
I20260925 07:33:31.284129 25600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:33:31.275382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.3:49077 (local address 127.9.66.62:35239)
0925 07:33:31.275448 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 07:33:31.275451 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:33:31.275644 (+ 193us) server_negotiation.cc:407] Connection header received
0925 07:33:31.276134 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:33:31.276136 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:33:31.276148 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:33:31.276207 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:33:31.277163 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:33:31.277715 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:33:31.278770 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:33:31.279057 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:33:31.279117 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:33:31.281445 (+ 2328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:33:31.281446 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:33:31.281446 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:33:31.281453 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:33:31.282991 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:33:31.283357 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:33:31.283358 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:33:31.283358 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:33:31.283417 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:33:31.283705 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:33:31.283705 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:33:31.283705 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:33:31.283788 (+ 83us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:33:31.283836 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 07:33:31.284018 (+ 182us) server_negotiation.cc:299] Negotiation successful
0925 07:33:31.284068 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260925 07:33:31.284381 25870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:33:31.275382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:35239 (local address 127.9.66.3:49077)
0925 07:33:31.275536 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 07:33:31.275569 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 07:33:31.276031 (+ 462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:33:31.276291 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:33:31.276295 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:33:31.276904 (+ 609us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:33:31.277072 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:33:31.277095 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:33:31.277830 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:33:31.277831 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:33:31.278693 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:33:31.278695 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:33:31.279149 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:33:31.279150 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:33:31.279285 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:33:31.279775 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:33:31.279786 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:33:31.281361 (+ 1575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:33:31.283074 (+ 1713us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:33:31.283080 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:33:31.283082 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:33:31.283291 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:33:31.283492 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:33:31.283493 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:33:31.283493 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:33:31.283638 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:33:31.283832 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:33:31.283834 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:33:31.283940 (+ 106us) client_negotiation.cc:769] Sending connection context
0925 07:33:31.284007 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 07:33:31.284121 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":76,"threads_started":1}
I20260925 07:33:31.284830 25868 heartbeater.cc:347] Connected to a master server at 127.9.66.62:35239
I20260925 07:33:31.284917 25868 heartbeater.cc:464] Registering TS with master...
I20260925 07:33:31.285038 25868 heartbeater.cc:511] Master 127.9.66.62:35239 requested a full tablet report, sending...
I20260925 07:33:31.285358 25409 ts_manager.cc:194] Registered new tserver with Master: 693a2a3014a34660ad9c23b17abceed7 (127.9.66.3:46457)
I20260925 07:33:31.285727 25409 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.66.3@KRBTEST.COM'} at 127.9.66.3:49077
I20260925 07:33:31.294265 9480 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 07:33:31.296592 25600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:33:31.294603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54088 (local address 127.9.66.62:35239)
0925 07:33:31.294656 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 07:33:31.294658 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:33:31.294718 (+ 60us) server_negotiation.cc:407] Connection header received
0925 07:33:31.294772 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:33:31.294773 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:33:31.294801 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:33:31.294857 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 07:33:31.294971 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:33:31.295402 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:33:31.296092 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:33:31.296255 (+ 163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:33:31.296302 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:33:31.296423 (+ 121us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 07:33:31.296457 (+ 34us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0925 07:33:31.296552 (+ 95us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54088: token expired
Metrics: {"server-negotiator.queue_time_us":32}
W20260925 07:33:31.296655 25600 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54088: token expired
W20260925 07:33:31.296787 25339 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.9.66.62:35239)
I20260925 07:33:31.302138 25600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:33:31.297062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54104 (local address 127.9.66.62:35239)
0925 07:33:31.297092 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0925 07:33:31.297093 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 07:33:31.297100 (+ 7us) server_negotiation.cc:407] Connection header received
0925 07:33:31.297137 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:33:31.297137 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:33:31.297144 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:33:31.297189 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:33:31.297642 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:33:31.298027 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:33:31.298609 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:33:31.298760 (+ 151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:33:31.298802 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:33:31.299859 (+ 1057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:33:31.299859 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:33:31.299860 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:33:31.299864 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:33:31.301202 (+ 1338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:33:31.301496 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:33:31.301496 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:33:31.301497 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:33:31.301617 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:33:31.301759 (+ 142us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:33:31.301760 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:33:31.301760 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:33:31.301867 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:33:31.301937 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 07:33:31.302032 (+ 95us) server_negotiation.cc:299] Negotiation successful
0925 07:33:31.302090 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260925 07:33:31.303099 25409 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54104:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 07:33:31.304783 25409 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 07:33:31.307233 25529 tablet_service.cc:1467] Processing CreateTablet for tablet 2fccfeef92d04be990f1091f1a284f62 (DEFAULT_TABLE table=test-table [id=46fd30bdc35c4194b69590f6494a2921]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:33:31.307449 25529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fccfeef92d04be990f1091f1a284f62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:33:31.308961 25489 tablet_bootstrap.cc:492] T 2fccfeef92d04be990f1091f1a284f62 P fb10411db4a0426fa59c5f405fbcaf03: Bootstrap starting.
I20260925 07:33:31.309278 25489 tablet_bootstrap.cc:654] T 2fccfeef92d04be990f1091f1a284f62 P fb10411db4a0426fa59c5f405fbcaf03: Neither blocks nor log segments found. Creating new log.
I20260925 07:33:31.309947 25489 tablet_bootstrap.cc:492] T 2fccfeef92d04be990f1091f1a284f62 P fb10411db4a0426fa59c5f405fbcaf03: No bootstrap required, opened a new log
I20260925 07:33:31.310001 25489 ts_tablet_manager.cc:1397] T 2fccfeef92d04be990f1091f1a284f62 P fb10411db4a0426fa59c5f405fbcaf03: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 07:33:31.310101 25489 raft_consensus.cc:348] T 2fccfeef92d04be990f1091f1a284f62 P fb10411db4a0426fa59c5f405fbcaf03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e2cf0a23c14e3e876406afcfdd6e07" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 33299 } } peers { permanent_uuid: "693a2a3014a34660ad9c23b17abceed7" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 46457 } } peers { permanent_uuid: "fb10411db4a0426fa59c5f405fbcaf03" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 34439 } }
I20260925 07:33:31.310173 25489 raft_consensus.cc:374] T 2fccfeef92d04be990f1091f1a284f62 P fb10411db4a0426fa59c5f405fbcaf03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:33:31.310225 25489 raft_consensus.cc:729] T 2fccfeef92d04be990f1091f1a284f62 P fb10411db4a0426fa59c5f405fbcaf03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb10411db4a0426fa59c5f405fbcaf03, State: Initialized, Role: FOLLOWER
I20260925 07:33:31.310278 25489 consensus_queue.cc:260] T 2fccfeef92d04be990f1091f1a284f62 P fb10411db4a0426fa59c5f405fbcaf03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e2cf0a23c14e3e876406afcfdd6e07" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 33299 } } peers { permanent_uuid: "693a2a3014a34660ad9c23b17abceed7" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 46457 } } peers { permanent_uuid: "fb10411db4a0426fa59c5f405fbcaf03" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 34439 } }
I20260925 07:33:31.310388 25489 ts_tablet_manager.cc:1428] T 2fccfeef92d04be990f1091f1a284f62 P fb10411db4a0426fa59c5f405fbcaf03: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 07:33:31.310503 25604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:33:31.306634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.2:33299 (local address 127.0.0.1:37112)
0925 07:33:31.306688 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0925 07:33:31.306697 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 07:33:31.306732 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:33:31.307159 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:33:31.307160 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:33:31.307162 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:33:31.307193 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:33:31.307195 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:33:31.308330 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:33:31.308331 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:33:31.309258 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:33:31.309260 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:33:31.310284 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:33:31.310285 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:33:31.310350 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:33:31.310354 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 07:33:31.310386 (+ 32us) client_negotiation.cc:240] Negotiation successful
0925 07:33:31.310444 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260925 07:33:31.311012 25874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:33:31.306730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.3:46457 (local address 127.0.0.1:45090)
0925 07:33:31.306905 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 07:33:31.306914 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 07:33:31.306955 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:33:31.307371 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:33:31.307372 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:33:31.307373 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:33:31.307407 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:33:31.307408 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:33:31.308728 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:33:31.308729 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:33:31.309791 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:33:31.309793 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:33:31.310772 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:33:31.310773 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:33:31.310858 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:33:31.310863 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 07:33:31.310909 (+ 46us) client_negotiation.cc:240] Negotiation successful
0925 07:33:31.310948 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":112,"threads_started":1}
I20260925 07:33:31.311029 25875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:33:31.306766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37112 (local address 127.9.66.2:33299)
0925 07:33:31.306948 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 07:33:31.306953 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 07:33:31.306963 (+ 10us) server_negotiation.cc:407] Connection header received
0925 07:33:31.307004 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:33:31.307006 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:33:31.307022 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:33:31.307267 (+ 245us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:33:31.307364 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:33:31.308249 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:33:31.309351 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:33:31.310179 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:33:31.310843 (+ 664us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:33:31.310900 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 07:33:31.310924 (+ 24us) server_negotiation.cc:299] Negotiation successful
0925 07:33:31.310971 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":122,"threads_started":1}
I20260925 07:33:31.311275 25876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:33:31.307026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45090 (local address 127.9.66.3:46457)
0925 07:33:31.307182 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0925 07:33:31.307185 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:33:31.307194 (+ 9us) server_negotiation.cc:407] Connection header received
0925 07:33:31.307227 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:33:31.307229 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:33:31.307242 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:33:31.307323 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:33:31.307474 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:33:31.308642 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:33:31.309882 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:33:31.310616 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:33:31.310682 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:33:31.310735 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 07:33:31.311128 (+ 393us) server_negotiation.cc:299] Negotiation successful
0925 07:33:31.311186 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":91,"threads_started":1}
I20260925 07:33:31.311522 25670 tablet_service.cc:1467] Processing CreateTablet for tablet 2fccfeef92d04be990f1091f1a284f62 (DEFAULT_TABLE table=test-table [id=46fd30bdc35c4194b69590f6494a2921]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:33:31.311777 25670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fccfeef92d04be990f1091f1a284f62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:33:31.311861 25802 tablet_service.cc:1467] Processing CreateTablet for tablet 2fccfeef92d04be990f1091f1a284f62 (DEFAULT_TABLE table=test-table [id=46fd30bdc35c4194b69590f6494a2921]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:33:31.312181 25802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fccfeef92d04be990f1091f1a284f62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:33:31.313555 25628 tablet_bootstrap.cc:492] T 2fccfeef92d04be990f1091f1a284f62 P a3e2cf0a23c14e3e876406afcfdd6e07: Bootstrap starting.
I20260925 07:33:31.314229 25628 tablet_bootstrap.cc:654] T 2fccfeef92d04be990f1091f1a284f62 P a3e2cf0a23c14e3e876406afcfdd6e07: Neither blocks nor log segments found. Creating new log.
I20260925 07:33:31.314517 25628 log.cc:824] T 2fccfeef92d04be990f1091f1a284f62 P a3e2cf0a23c14e3e876406afcfdd6e07: Log is configured to *not* fsync() on all Append() calls
I20260925 07:33:31.315207 25628 tablet_bootstrap.cc:492] T 2fccfeef92d04be990f1091f1a284f62 P a3e2cf0a23c14e3e876406afcfdd6e07: No bootstrap required, opened a new log
I20260925 07:33:31.315275 25628 ts_tablet_manager.cc:1397] T 2fccfeef92d04be990f1091f1a284f62 P a3e2cf0a23c14e3e876406afcfdd6e07: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 07:33:31.316512 25628 raft_consensus.cc:348] T 2fccfeef92d04be990f1091f1a284f62 P a3e2cf0a23c14e3e876406afcfdd6e07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e2cf0a23c14e3e876406afcfdd6e07" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 33299 } } peers { permanent_uuid: "693a2a3014a34660ad9c23b17abceed7" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 46457 } } peers { permanent_uuid: "fb10411db4a0426fa59c5f405fbcaf03" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 34439 } }
I20260925 07:33:31.316676 25628 raft_consensus.cc:374] T 2fccfeef92d04be990f1091f1a284f62 P a3e2cf0a23c14e3e876406afcfdd6e07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:33:31.316720 25628 raft_consensus.cc:729] T 2fccfeef92d04be990f1091f1a284f62 P a3e2cf0a23c14e3e876406afcfdd6e07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3e2cf0a23c14e3e876406afcfdd6e07, State: Initialized, Role: FOLLOWER
I20260925 07:33:31.316833 25628 consensus_queue.cc:260] T 2fccfeef92d04be990f1091f1a284f62 P a3e2cf0a23c14e3e876406afcfdd6e07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e2cf0a23c14e3e876406afcfdd6e07" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 33299 } } peers { permanent_uuid: "693a2a3014a34660ad9c23b17abceed7" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 46457 } } peers { permanent_uuid: "fb10411db4a0426fa59c5f405fbcaf03" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 34439 } }
I20260925 07:33:31.317137 25736 heartbeater.cc:503] Master 127.9.66.62:35239 was elected leader, sending a full tablet report...
I20260925 07:33:31.317122 25628 ts_tablet_manager.cc:1428] T 2fccfeef92d04be990f1091f1a284f62 P a3e2cf0a23c14e3e876406afcfdd6e07: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 07:33:31.317243 25879 tablet_bootstrap.cc:492] T 2fccfeef92d04be990f1091f1a284f62 P 693a2a3014a34660ad9c23b17abceed7: Bootstrap starting.
I20260925 07:33:31.317901 25879 tablet_bootstrap.cc:654] T 2fccfeef92d04be990f1091f1a284f62 P 693a2a3014a34660ad9c23b17abceed7: Neither blocks nor log segments found. Creating new log.
I20260925 07:33:31.318506 25879 log.cc:824] T 2fccfeef92d04be990f1091f1a284f62 P 693a2a3014a34660ad9c23b17abceed7: Log is configured to *not* fsync() on all Append() calls
I20260925 07:33:31.319164 25879 tablet_bootstrap.cc:492] T 2fccfeef92d04be990f1091f1a284f62 P 693a2a3014a34660ad9c23b17abceed7: No bootstrap required, opened a new log
I20260925 07:33:31.319231 25879 ts_tablet_manager.cc:1397] T 2fccfeef92d04be990f1091f1a284f62 P 693a2a3014a34660ad9c23b17abceed7: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260925 07:33:31.320511 25879 raft_consensus.cc:348] T 2fccfeef92d04be990f1091f1a284f62 P 693a2a3014a34660ad9c23b17abceed7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e2cf0a23c14e3e876406afcfdd6e07" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 33299 } } peers { permanent_uuid: "693a2a3014a34660ad9c23b17abceed7" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 46457 } } peers { permanent_uuid: "fb10411db4a0426fa59c5f405fbcaf03" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 34439 } }
I20260925 07:33:31.320637 25879 raft_consensus.cc:374] T 2fccfeef92d04be990f1091f1a284f62 P 693a2a3014a34660ad9c23b17abceed7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:33:31.320681 25879 raft_consensus.cc:729] T 2fccfeef92d04be990f1091f1a284f62 P 693a2a3014a34660ad9c23b17abceed7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 693a2a3014a34660ad9c23b17abceed7, State: Initialized, Role: FOLLOWER
I20260925 07:33:31.320789 25879 consensus_queue.cc:260] T 2fccfeef92d04be990f1091f1a284f62 P 693a2a3014a34660ad9c23b17abceed7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e2cf0a23c14e3e876406afcfdd6e07" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 33299 } } peers { permanent_uuid: "693a2a3014a34660ad9c23b17abceed7" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 46457 } } peers { permanent_uuid: "fb10411db4a0426fa59c5f405fbcaf03" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 34439 } }
I20260925 07:33:31.321059 25879 ts_tablet_manager.cc:1428] T 2fccfeef92d04be990f1091f1a284f62 P 693a2a3014a34660ad9c23b17abceed7: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260925 07:33:31.321090 25868 heartbeater.cc:503] Master 127.9.66.62:35239 was elected leader, sending a full tablet report...
W20260925 07:33:31.395185 25737 tablet.cc:2369] T 2fccfeef92d04be990f1091f1a284f62 P a3e2cf0a23c14e3e876406afcfdd6e07: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 07:33:31.502187 25602 raft_consensus.cc:482] T 2fccfeef92d04be990f1091f1a284f62 P fb10411db4a0426fa59c5f405fbcaf03 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 07:33:31.502311 25602 raft_consensus.cc:504] T 2fccfeef92d04be990f1091f1a284f62 P fb10411db4a0426fa59c5f405fbcaf03 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e2cf0a23c14e3e876406afcfdd6e07" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 33299 } } peers { permanent_uuid: "693a2a3014a34660ad9c23b17abceed7" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 46457 } } peers { permanent_uuid: "fb10411db4a0426fa59c5f405fbcaf03" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 34439 } }
I20260925 07:33:31.502599 25602 leader_election.cc:288] T 2fccfeef92d04be990f1091f1a284f62 P fb10411db4a0426fa59c5f405fbcaf03 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a3e2cf0a23c14e3e876406afcfdd6e07 (127.9.66.2:33299), 693a2a3014a34660ad9c23b17abceed7 (127.9.66.3:46457)
I20260925 07:33:31.505908 25599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:33:31.502778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.2:33299 (local address 127.9.66.1:50605)
0925 07:33:31.502830 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0925 07:33:31.502841 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 07:33:31.502888 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:33:31.503090 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:33:31.503091 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:33:31.503093 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:33:31.503171 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:33:31.503172 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:33:31.503784 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:33:31.503784 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:33:31.504836 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:33:31.504839 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:33:31.505634 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:33:31.505635 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:33:31.505706 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:33:31.505710 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 07:33:31.505779 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 07:33:31.505816 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260925 07:33:31.505931 25875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:33:31.502889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.1:50605 (local address 127.9.66.2:33299)
0925 07:33:31.502946 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 07:33:31.502948 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:33:31.502957 (+ 9us) server_negotiation.cc:407] Connection header received
0925 07:33:31.502991 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:33:31.502992 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:33:31.503002 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:33:31.503057 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:33:31.503243 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:33:31.503713 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:33:31.504919 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:33:31.505558 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:33:31.505636 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:33:31.505646 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 07:33:31.505831 (+ 185us) server_negotiation.cc:299] Negotiation successful
0925 07:33:31.505881 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260925 07:33:31.506239 25690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fccfeef92d04be990f1091f1a284f62" candidate_uuid: "fb10411db4a0426fa59c5f405fbcaf03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3e2cf0a23c14e3e876406afcfdd6e07" is_pre_election: true
I20260925 07:33:31.506402 25690 raft_consensus.cc:2454] T 2fccfeef92d04be990f1091f1a284f62 P a3e2cf0a23c14e3e876406afcfdd6e07 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb10411db4a0426fa59c5f405fbcaf03 in term 0.
I20260925 07:33:31.506635 25883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:33:31.502897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.3:46457 (local address 127.9.66.1:54213)
0925 07:33:31.502980 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0925 07:33:31.502988 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 07:33:31.503025 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:33:31.503294 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:33:31.503295 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:33:31.503296 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:33:31.503598 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:33:31.503600 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:33:31.504471 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:33:31.504471 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:33:31.505648 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:33:31.505653 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:33:31.506369 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:33:31.506370 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:33:31.506439 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:33:31.506442 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 07:33:31.506537 (+ 95us) client_negotiation.cc:240] Negotiation successful
0925 07:33:31.506570 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73,"thread_start_us":52,"threads_started":1}
I20260925 07:33:31.506624 25484 leader_election.cc:302] T 2fccfeef92d04be990f1091f1a284f62 P fb10411db4a0426fa59c5f405fbcaf03 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a3e2cf0a23c14e3e876406afcfdd6e07, fb10411db4a0426fa59c5f405fbcaf03; no voters:
I20260925 07:33:31.506765 25876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:33:31.503129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.1:54213 (local address 127.9.66.3:46457)
0925 07:33:31.503192 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 07:33:31.503194 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:33:31.503202 (+ 8us) server_negotiation.cc:407] Connection header received
0925 07:33:31.503226 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:33:31.503227 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:33:31.503240 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:33:31.503284 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:33:31.503674 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:33:31.504370 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:33:31.505720 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:33:31.506281 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:33:31.506351 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:33:31.506360 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 07:33:31.506665 (+ 305us) server_negotiation.cc:299] Negotiation successful
0925 07:33:31.506706 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 07:33:31.506779 25602 raft_consensus.cc:2781] T 2fccfeef92d04be990f1091f1a284f62 P fb10411db4a0426fa59c5f405fbcaf03 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 07:33:31.506829 25602 raft_consensus.cc:482] T 2fccfeef92d04be990f1091f1a284f62 P fb10411db4a0426fa59c5f405fbcaf03 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 07:33:31.506848 25602 raft_consensus.cc:3037] T 2fccfeef92d04be990f1091f1a284f62 P fb10411db4a0426fa59c5f405fbcaf03 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:33:31.507025 25822 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fccfeef92d04be990f1091f1a284f62" candidate_uuid: "fb10411db4a0426fa59c5f405fbcaf03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "693a2a3014a34660ad9c23b17abceed7" is_pre_election: true
I20260925 07:33:31.507190 25822 raft_consensus.cc:2454] T 2fccfeef92d04be990f1091f1a284f62 P 693a2a3014a34660ad9c23b17abceed7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb10411db4a0426fa59c5f405fbcaf03 in term 0.
I20260925 07:33:31.507673 25602 raft_consensus.cc:504] T 2fccfeef92d04be990f1091f1a284f62 P fb10411db4a0426fa59c5f405fbcaf03 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e2cf0a23c14e3e876406afcfdd6e07" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 33299 } } peers { permanent_uuid: "693a2a3014a34660ad9c23b17abceed7" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 46457 } } peers { permanent_uuid: "fb10411db4a0426fa59c5f405fbcaf03" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 34439 } }
I20260925 07:33:31.507812 25602 leader_election.cc:288] T 2fccfeef92d04be990f1091f1a284f62 P fb10411db4a0426fa59c5f405fbcaf03 [CANDIDATE]: Term 1 election: Requested vote from peers a3e2cf0a23c14e3e876406afcfdd6e07 (127.9.66.2:33299), 693a2a3014a34660ad9c23b17abceed7 (127.9.66.3:46457)
I20260925 07:33:31.507948 25690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fccfeef92d04be990f1091f1a284f62" candidate_uuid: "fb10411db4a0426fa59c5f405fbcaf03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3e2cf0a23c14e3e876406afcfdd6e07"
I20260925 07:33:31.507992 25822 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fccfeef92d04be990f1091f1a284f62" candidate_uuid: "fb10411db4a0426fa59c5f405fbcaf03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "693a2a3014a34660ad9c23b17abceed7"
I20260925 07:33:31.508028 25690 raft_consensus.cc:3037] T 2fccfeef92d04be990f1091f1a284f62 P a3e2cf0a23c14e3e876406afcfdd6e07 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:33:31.508045 25822 raft_consensus.cc:3037] T 2fccfeef92d04be990f1091f1a284f62 P 693a2a3014a34660ad9c23b17abceed7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:33:31.508731 25822 raft_consensus.cc:2454] T 2fccfeef92d04be990f1091f1a284f62 P 693a2a3014a34660ad9c23b17abceed7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb10411db4a0426fa59c5f405fbcaf03 in term 1.
I20260925 07:33:31.508735 25690 raft_consensus.cc:2454] T 2fccfeef92d04be990f1091f1a284f62 P a3e2cf0a23c14e3e876406afcfdd6e07 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb10411db4a0426fa59c5f405fbcaf03 in term 1.
I20260925 07:33:31.508880 25484 leader_election.cc:302] T 2fccfeef92d04be990f1091f1a284f62 P fb10411db4a0426fa59c5f405fbcaf03 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a3e2cf0a23c14e3e876406afcfdd6e07, fb10411db4a0426fa59c5f405fbcaf03; no voters:
I20260925 07:33:31.508957 25602 raft_consensus.cc:2781] T 2fccfeef92d04be990f1091f1a284f62 P fb10411db4a0426fa59c5f405fbcaf03 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:33:31.509117 25602 raft_consensus.cc:686] T 2fccfeef92d04be990f1091f1a284f62 P fb10411db4a0426fa59c5f405fbcaf03 [term 1 LEADER]: Becoming Leader. State: Replica: fb10411db4a0426fa59c5f405fbcaf03, State: Running, Role: LEADER
I20260925 07:33:31.509189 25602 consensus_queue.cc:237] T 2fccfeef92d04be990f1091f1a284f62 P fb10411db4a0426fa59c5f405fbcaf03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e2cf0a23c14e3e876406afcfdd6e07" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 33299 } } peers { permanent_uuid: "693a2a3014a34660ad9c23b17abceed7" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 46457 } } peers { permanent_uuid: "fb10411db4a0426fa59c5f405fbcaf03" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 34439 } }
I20260925 07:33:31.509953 25409 catalog_manager.cc:5495] T 2fccfeef92d04be990f1091f1a284f62 P fb10411db4a0426fa59c5f405fbcaf03 reported cstate change: term changed from 0 to 1, leader changed from <none> to fb10411db4a0426fa59c5f405fbcaf03 (127.9.66.1). New cstate: current_term: 1 leader_uuid: "fb10411db4a0426fa59c5f405fbcaf03" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e2cf0a23c14e3e876406afcfdd6e07" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 33299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "693a2a3014a34660ad9c23b17abceed7" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 46457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb10411db4a0426fa59c5f405fbcaf03" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 34439 } health_report { overall_health: HEALTHY } } }
W20260925 07:33:31.519527 25598 tablet.cc:2369] T 2fccfeef92d04be990f1091f1a284f62 P fb10411db4a0426fa59c5f405fbcaf03: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 07:33:31.525231 25869 tablet.cc:2369] T 2fccfeef92d04be990f1091f1a284f62 P 693a2a3014a34660ad9c23b17abceed7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 07:33:31.535720 25605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:33:31.533701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59010 (local address 127.9.66.1:34439)
0925 07:33:31.533762 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 07:33:31.533763 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 07:33:31.533771 (+ 8us) server_negotiation.cc:407] Connection header received
0925 07:33:31.533799 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:33:31.533800 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:33:31.533816 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:33:31.533875 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 07:33:31.534009 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:33:31.534458 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:33:31.535129 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:33:31.535312 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:33:31.535358 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:33:31.535481 (+ 123us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 07:33:31.535567 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 07:33:31.535616 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 07:33:31.535656 (+ 40us) server_negotiation.cc:299] Negotiation successful
0925 07:33:31.535673 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 07:33:31.537159 25690 raft_consensus.cc:1261] T 2fccfeef92d04be990f1091f1a284f62 P a3e2cf0a23c14e3e876406afcfdd6e07 [term 1 FOLLOWER]: Refusing update from remote peer fb10411db4a0426fa59c5f405fbcaf03: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:33:31.537159 25822 raft_consensus.cc:1261] T 2fccfeef92d04be990f1091f1a284f62 P 693a2a3014a34660ad9c23b17abceed7 [term 1 FOLLOWER]: Refusing update from remote peer fb10411db4a0426fa59c5f405fbcaf03: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:33:31.537493 25884 consensus_queue.cc:1035] T 2fccfeef92d04be990f1091f1a284f62 P fb10411db4a0426fa59c5f405fbcaf03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3e2cf0a23c14e3e876406afcfdd6e07" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 33299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:33:31.537619 25602 consensus_queue.cc:1035] T 2fccfeef92d04be990f1091f1a284f62 P fb10411db4a0426fa59c5f405fbcaf03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "693a2a3014a34660ad9c23b17abceed7" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 46457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:33:31.539835 25887 mvcc.cc:205] Tried to move back new op lower bound from 7333157320853581824 to 7333157320742260736. Current Snapshot: MvccSnapshot[applied={T|T < 7333157320853581824}]
I20260925 07:33:31.540349 25889 mvcc.cc:205] Tried to move back new op lower bound from 7333157320853581824 to 7333157320742260736. Current Snapshot: MvccSnapshot[applied={T|T < 7333157320853581824}]
I20260925 07:33:31.541819 25409 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54104:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 07:33:31.541913 25409 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54104:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 07:33:31.542709 25409 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 10c9b33b17904def90c291c5a7091209: Sending DeleteTablet for 3 replicas of tablet 2fccfeef92d04be990f1091f1a284f62
I20260925 07:33:31.542999 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 25447
I20260925 07:33:31.543067 25670 tablet_service.cc:1514] Processing DeleteTablet for tablet 2fccfeef92d04be990f1091f1a284f62 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:33:31 UTC) from {username='kudu', principal='kudu/127.9.66.62@KRBTEST.COM'} at 127.0.0.1:37112
I20260925 07:33:31.543090 25529 tablet_service.cc:1514] Processing DeleteTablet for tablet 2fccfeef92d04be990f1091f1a284f62 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:33:31 UTC) from {username='kudu', principal='kudu/127.9.66.62@KRBTEST.COM'} at 127.0.0.1:59004
I20260925 07:33:31.543141 25801 tablet_service.cc:1514] Processing DeleteTablet for tablet 2fccfeef92d04be990f1091f1a284f62 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:33:31 UTC) from {username='kudu', principal='kudu/127.9.66.62@KRBTEST.COM'} at 127.0.0.1:45090
I20260925 07:33:31.543232 25606 tablet_replica.cc:330] T 2fccfeef92d04be990f1091f1a284f62 P fb10411db4a0426fa59c5f405fbcaf03: stopping tablet replica
I20260925 07:33:31.543291 25606 raft_consensus.cc:2229] T 2fccfeef92d04be990f1091f1a284f62 P fb10411db4a0426fa59c5f405fbcaf03 [term 1 LEADER]: Raft consensus shutting down.
I20260925 07:33:31.543363 25606 raft_consensus.cc:2258] T 2fccfeef92d04be990f1091f1a284f62 P fb10411db4a0426fa59c5f405fbcaf03 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:33:31.543516 25802 tablet_service.cc:1514] Processing DeleteTablet for tablet 2fccfeef92d04be990f1091f1a284f62 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:33:31 UTC) from {username='kudu', principal='kudu/127.9.66.62@KRBTEST.COM'} at 127.0.0.1:45090
I20260925 07:33:31.543622 25897 tablet_replica.cc:330] T 2fccfeef92d04be990f1091f1a284f62 P 693a2a3014a34660ad9c23b17abceed7: stopping tablet replica
I20260925 07:33:31.543702 25897 raft_consensus.cc:2229] T 2fccfeef92d04be990f1091f1a284f62 P 693a2a3014a34660ad9c23b17abceed7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 07:33:31.543793 25897 raft_consensus.cc:2258] T 2fccfeef92d04be990f1091f1a284f62 P 693a2a3014a34660ad9c23b17abceed7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:33:31.544201 25897 ts_tablet_manager.cc:1905] T 2fccfeef92d04be990f1091f1a284f62 P 693a2a3014a34660ad9c23b17abceed7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 07:33:31.546114 25897 ts_tablet_manager.cc:1918] T 2fccfeef92d04be990f1091f1a284f62 P 693a2a3014a34660ad9c23b17abceed7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 07:33:31.546180 25897 log.cc:1196] T 2fccfeef92d04be990f1091f1a284f62 P 693a2a3014a34660ad9c23b17abceed7: Deleting WAL directory at /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-2/wal/wals/2fccfeef92d04be990f1091f1a284f62
I20260925 07:33:31.546449 25897 ts_tablet_manager.cc:1939] T 2fccfeef92d04be990f1091f1a284f62 P 693a2a3014a34660ad9c23b17abceed7: Deleting consensus metadata
I20260925 07:33:31.546856 25394 catalog_manager.cc:4841] TS 693a2a3014a34660ad9c23b17abceed7 (127.9.66.3:46457): tablet 2fccfeef92d04be990f1091f1a284f62 (table test-table [id=46fd30bdc35c4194b69590f6494a2921]) successfully deleted
W20260925 07:33:31.547027 25394 catalog_manager.cc:4808] TS 693a2a3014a34660ad9c23b17abceed7 (127.9.66.3:46457): delete failed for tablet 2fccfeef92d04be990f1091f1a284f62 because the tablet was not found. No further retry: Not found: Tablet not found: 2fccfeef92d04be990f1091f1a284f62
W20260925 07:33:31.549616 25394 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.66.1:34439 (error 108)
I20260925 07:33:31.549896 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 25603
I20260925 07:33:31.550114 25874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:33:31.549978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.1:34439 (local address 127.0.0.1:59022)
0925 07:33:31.550044 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0925 07:33:31.550079 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.1:34439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260925 07:33:31.550617 25394 catalog_manager.cc:4584] TS fb10411db4a0426fa59c5f405fbcaf03 (127.9.66.1:34439): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2fccfeef92d04be990f1091f1a284f62: Network error: Client connection negotiation failed: client connection to 127.9.66.1:34439: connect: Connection refused (error 111)
I20260925 07:33:31.554876 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 25739
I20260925 07:33:31.554910 25874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:33:31.554786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.2:33299 (local address 127.0.0.1:37116)
0925 07:33:31.554838 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0925 07:33:31.554887 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.2:33299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260925 07:33:31.559604 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 25376
I20260925 07:33:51.564949 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.66.62:35239
--webserver_interface=127.9.66.62
--webserver_port=46119
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab
--principal=kudu/127.9.66.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:43203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.66.62:35239
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:33:51.644582 25899 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:33:51.644763 25899 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:33:51.644785 25899 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:33:51.645946 25899 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 07:33:51.645984 25899 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 07:33:51.645996 25899 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 07:33:51.646008 25899 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:33:51.646056 25899 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:33:51.646075 25899 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:33:51.646097 25899 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:33:51.646121 25899 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:33:51.647560 25899 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:43203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.9.66.62:35239
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab
--principal=kudu/127.9.66.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.66.62:35239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.66.62
--webserver_port=46119
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25899
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:33:51.647786 25899 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:33:51.648026 25899 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:33:51.653126 25907 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:33:51.653151 25899 server_base.cc:1034] running on GCE node
W20260925 07:33:51.653115 25905 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:33:51.653106 25904 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:33:51.653635 25899 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:33:51.653960 25899 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:33:51.655133 25899 hybrid_clock.cc:648] HybridClock initialized: now 1790321631655111 us; error 34 us; skew 500 ppm
Sep 25 07:33:51 dist-test-slave-57fn krb5kdc[24831](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321631, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:33:51.657612 25899 init.cc:377] Logged in from keytab as kudu/127.9.66.62@KRBTEST.COM (short username kudu)
I20260925 07:33:51.658032 25899 webserver.cc:460] Webserver started at http://127.9.66.62:46119/ using document root <none> and password file <none>
I20260925 07:33:51.658257 25899 fs_manager.cc:359] Metadata directory not provided
I20260925 07:33:51.658324 25899 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:33:51.659555 25899 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:33:51.660207 25914 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:33:51.660377 25899 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 07:33:51.660471 25899 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/master-0/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/master-0/wal
uuid: "10c9b33b17904def90c291c5a7091209"
format_stamp: "Formatted at 2026-09-25 07:33:09 on dist-test-slave-57fn"
server_key: "41bae02c09abafa675bad2414d6b6471"
server_key_iv: "7b491795d817902dbd50efa44281d65e"
server_key_version: "encryptionkey@0"
I20260925 07:33:51.660776 25899 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:33:51.674170 25899 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:33:51.675966 25899 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:33:51.676137 25899 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:33:51.680049 25899 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.62:35239
I20260925 07:33:51.680078 25966 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.62:35239 every 8 connection(s)
I20260925 07:33:51.680436 25899 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
I20260925 07:33:51.681885 25967 sys_catalog.cc:263] Verifying existing consensus state
I20260925 07:33:51.682528 25967 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10c9b33b17904def90c291c5a7091209: Bootstrap starting.
I20260925 07:33:51.689930 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 25899
I20260925 07:33:51.690201 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.66.1:34439
--local_ip_for_outbound_sockets=127.9.66.1
--tserver_master_addrs=127.9.66.62:35239
--webserver_port=32927
--webserver_interface=127.9.66.1
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.1.keytab
--principal=kudu/127.9.66.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:43203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 07:33:51.690726 25967 log.cc:824] T 00000000000000000000000000000000 P 10c9b33b17904def90c291c5a7091209: Log is configured to *not* fsync() on all Append() calls
I20260925 07:33:51.693193 25967 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10c9b33b17904def90c291c5a7091209: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 07:33:51.693539 25967 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10c9b33b17904def90c291c5a7091209: Bootstrap complete.
I20260925 07:33:51.697412 25967 raft_consensus.cc:348] T 00000000000000000000000000000000 P 10c9b33b17904def90c291c5a7091209 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10c9b33b17904def90c291c5a7091209" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 35239 } }
I20260925 07:33:51.697798 25967 raft_consensus.cc:729] T 00000000000000000000000000000000 P 10c9b33b17904def90c291c5a7091209 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10c9b33b17904def90c291c5a7091209, State: Initialized, Role: FOLLOWER
I20260925 07:33:51.697978 25967 consensus_queue.cc:260] T 00000000000000000000000000000000 P 10c9b33b17904def90c291c5a7091209 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10c9b33b17904def90c291c5a7091209" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 35239 } }
I20260925 07:33:51.698063 25967 raft_consensus.cc:388] T 00000000000000000000000000000000 P 10c9b33b17904def90c291c5a7091209 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 07:33:51.698108 25967 raft_consensus.cc:482] T 00000000000000000000000000000000 P 10c9b33b17904def90c291c5a7091209 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 07:33:51.698164 25967 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 10c9b33b17904def90c291c5a7091209 [term 2 FOLLOWER]: Advancing to term 3
I20260925 07:33:51.699666 25967 raft_consensus.cc:504] T 00000000000000000000000000000000 P 10c9b33b17904def90c291c5a7091209 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10c9b33b17904def90c291c5a7091209" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 35239 } }
I20260925 07:33:51.699786 25967 leader_election.cc:302] T 00000000000000000000000000000000 P 10c9b33b17904def90c291c5a7091209 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 10c9b33b17904def90c291c5a7091209; no voters:
I20260925 07:33:51.699976 25967 leader_election.cc:288] T 00000000000000000000000000000000 P 10c9b33b17904def90c291c5a7091209 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260925 07:33:51.700043 25971 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 10c9b33b17904def90c291c5a7091209 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 07:33:51.700192 25971 raft_consensus.cc:686] T 00000000000000000000000000000000 P 10c9b33b17904def90c291c5a7091209 [term 3 LEADER]: Becoming Leader. State: Replica: 10c9b33b17904def90c291c5a7091209, State: Running, Role: LEADER
I20260925 07:33:51.700241 25967 sys_catalog.cc:564] T 00000000000000000000000000000000 P 10c9b33b17904def90c291c5a7091209 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:33:51.700336 25971 consensus_queue.cc:237] T 00000000000000000000000000000000 P 10c9b33b17904def90c291c5a7091209 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10c9b33b17904def90c291c5a7091209" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 35239 } }
I20260925 07:33:51.700737 25973 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10c9b33b17904def90c291c5a7091209 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 10c9b33b17904def90c291c5a7091209. Latest consensus state: current_term: 3 leader_uuid: "10c9b33b17904def90c291c5a7091209" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10c9b33b17904def90c291c5a7091209" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 35239 } } }
I20260925 07:33:51.700739 25972 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10c9b33b17904def90c291c5a7091209 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "10c9b33b17904def90c291c5a7091209" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10c9b33b17904def90c291c5a7091209" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 35239 } } }
I20260925 07:33:51.700850 25973 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10c9b33b17904def90c291c5a7091209 [sys.catalog]: This master's current role is: LEADER
I20260925 07:33:51.700853 25972 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10c9b33b17904def90c291c5a7091209 [sys.catalog]: This master's current role is: LEADER
I20260925 07:33:51.701426 25979 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 07:33:51.702229 25979 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 07:33:51.702525 25979 catalog_manager.cc:1259] Loaded cluster ID: dd1c0386aa0144abaed6dc16b9d4fffc
I20260925 07:33:51.702566 25979 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 07:33:51.703337 25979 catalog_manager.cc:1504] Loading token signing keys...
I20260925 07:33:51.703529 25979 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 10c9b33b17904def90c291c5a7091209: Loaded TSK: 0
I20260925 07:33:51.703586 25979 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 10c9b33b17904def90c291c5a7091209: Loaded TSK: 1
F20260925 07:33:51.703814 25979 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fdc881d69ad google::LogMessage::Fail() at ??:0
@ 0x7fdc881d988a google::LogMessage::SendToLog() at ??:0
@ 0x7fdc881d64de google::LogMessage::Flush() at ??:0
@ 0x7fdc881d7a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fdc8a943104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fdc8a940d41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fdc901301cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fdc9013048e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fdc90130594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fdc900f6c48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fdc90100ad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fdc8a0a40d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fdc8a099954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fdc8aefd6db start_thread at ??:0
@ 0x7fdc86cd461f clone at ??:0
W20260925 07:33:51.787127 25970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:33:51.787343 25970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:33:51.787387 25970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:33:51.788508 25970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:33:51.788571 25970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:33:51.788617 25970 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.66.1
I20260925 07:33:51.790105 25970 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:43203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.1.keytab
--principal=kudu/127.9.66.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.66.1:34439
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.66.1
--webserver_port=32927
--tserver_master_addrs=127.9.66.62:35239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25970
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.66.1
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:33:51.790372 25970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:33:51.790588 25970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:33:51.795630 26009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:33:51.795696 26008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:33:51.795619 26011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:33:51.796275 25970 server_base.cc:1034] running on GCE node
I20260925 07:33:51.796514 25970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:33:51.796749 25970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:33:51.797910 25970 hybrid_clock.cc:648] HybridClock initialized: now 1790321631797897 us; error 36 us; skew 500 ppm
Sep 25 07:33:51 dist-test-slave-57fn krb5kdc[24831](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321631, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:33:51.800387 25970 init.cc:377] Logged in from keytab as kudu/127.9.66.1@KRBTEST.COM (short username kudu)
I20260925 07:33:51.800809 25970 webserver.cc:460] Webserver started at http://127.9.66.1:32927/ using document root <none> and password file <none>
I20260925 07:33:51.801045 25970 fs_manager.cc:359] Metadata directory not provided
I20260925 07:33:51.801113 25970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:33:51.802547 25970 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:33:51.803277 26018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:33:51.803469 25970 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:33:51.803573 25970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-0/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-0/wal
uuid: "fb10411db4a0426fa59c5f405fbcaf03"
format_stamp: "Formatted at 2026-09-25 07:33:10 on dist-test-slave-57fn"
server_key: "896b5afaa9608339fc8459e883be320b"
server_key_iv: "5adf2d1d9de966513d2a996252efe614"
server_key_version: "encryptionkey@0"
I20260925 07:33:51.803908 25970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:33:51.820034 25970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:33:51.822290 25970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:33:51.822468 25970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:33:51.822692 25970 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:33:51.823135 26025 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 07:33:51.825062 25970 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 07:33:51.825125 25970 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260925 07:33:51.825170 25970 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 07:33:51.825878 25970 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 07:33:51.825930 25970 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.001s sys 0.000s
I20260925 07:33:51.825999 26025 tablet_bootstrap.cc:492] T 2fccfeef92d04be990f1091f1a284f62 P fb10411db4a0426fa59c5f405fbcaf03: Bootstrap starting.
I20260925 07:33:51.832873 25970 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.1:34439
I20260925 07:33:51.832955 26132 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.1:34439 every 8 connection(s)
I20260925 07:33:51.833280 25970 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-0/data/info.pb
I20260925 07:33:51.835839 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 25970
I20260925 07:33:51.836124 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.66.2:33299
--local_ip_for_outbound_sockets=127.9.66.2
--tserver_master_addrs=127.9.66.62:35239
--webserver_port=34779
--webserver_interface=127.9.66.2
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.2.keytab
--principal=kudu/127.9.66.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:43203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 07:33:51.836942 26025 log.cc:824] T 2fccfeef92d04be990f1091f1a284f62 P fb10411db4a0426fa59c5f405fbcaf03: Log is configured to *not* fsync() on all Append() calls
I20260925 07:33:51.839589 26025 tablet_bootstrap.cc:492] T 2fccfeef92d04be990f1091f1a284f62 P fb10411db4a0426fa59c5f405fbcaf03: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 07:33:51.839890 26025 tablet_bootstrap.cc:492] T 2fccfeef92d04be990f1091f1a284f62 P fb10411db4a0426fa59c5f405fbcaf03: Bootstrap complete.
I20260925 07:33:51.841071 26025 ts_tablet_manager.cc:1397] T 2fccfeef92d04be990f1091f1a284f62 P fb10411db4a0426fa59c5f405fbcaf03: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.013s
I20260925 07:33:51.842465 26025 raft_consensus.cc:348] T 2fccfeef92d04be990f1091f1a284f62 P fb10411db4a0426fa59c5f405fbcaf03 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e2cf0a23c14e3e876406afcfdd6e07" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 33299 } } peers { permanent_uuid: "693a2a3014a34660ad9c23b17abceed7" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 46457 } } peers { permanent_uuid: "fb10411db4a0426fa59c5f405fbcaf03" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 34439 } }
I20260925 07:33:51.842796 26025 raft_consensus.cc:729] T 2fccfeef92d04be990f1091f1a284f62 P fb10411db4a0426fa59c5f405fbcaf03 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb10411db4a0426fa59c5f405fbcaf03, State: Initialized, Role: FOLLOWER
I20260925 07:33:51.842967 26025 consensus_queue.cc:260] T 2fccfeef92d04be990f1091f1a284f62 P fb10411db4a0426fa59c5f405fbcaf03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e2cf0a23c14e3e876406afcfdd6e07" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 33299 } } peers { permanent_uuid: "693a2a3014a34660ad9c23b17abceed7" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 46457 } } peers { permanent_uuid: "fb10411db4a0426fa59c5f405fbcaf03" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 34439 } }
I20260925 07:33:51.843233 26025 ts_tablet_manager.cc:1428] T 2fccfeef92d04be990f1091f1a284f62 P fb10411db4a0426fa59c5f405fbcaf03: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
W20260925 07:33:51.903221 26135 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:33:51.833929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:35239 (local address 127.9.66.1:47589)
0925 07:33:51.834095 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 07:33:51.834134 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 07:33:51.834622 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:33:51.903022 (+ 68400us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:35239: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":94,"threads_started":1}
W20260925 07:33:51.903465 26133 heartbeater.cc:644] Failed to heartbeat to 127.9.66.62:35239 (0 consecutive failures): Network error: Failed to ping master at 127.9.66.62:35239: Client connection negotiation failed: client connection to 127.9.66.62:35239: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 07:33:51.903810 26135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:33:51.903686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:35239 (local address 127.9.66.1:34365)
0925 07:33:51.903741 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0925 07:33:51.903787 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:35239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260925 07:33:51.904227 26135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:33:51.904135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:35239 (local address 127.9.66.1:35369)
0925 07:33:51.904183 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0925 07:33:51.904210 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:35239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260925 07:33:51.904362 26133 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 07:33:51.919927 26137 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:33:51.920156 26137 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:33:51.920205 26137 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:33:51.921373 26137 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:33:51.921434 26137 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:33:51.921478 26137 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.66.2
I20260925 07:33:51.922983 26137 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:43203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.2.keytab
--principal=kudu/127.9.66.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.66.2:33299
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.66.2
--webserver_port=34779
--tserver_master_addrs=127.9.66.62:35239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26137
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.66.2
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:33:51.923236 26137 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:33:51.923460 26137 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:33:51.928390 26143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:33:51.928364 26144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:33:51.928560 26137 server_base.cc:1034] running on GCE node
W20260925 07:33:51.928378 26146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:33:51.928762 26137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:33:51.928988 26137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:33:51.930181 26137 hybrid_clock.cc:648] HybridClock initialized: now 1790321631930173 us; error 35 us; skew 500 ppm
Sep 25 07:33:51 dist-test-slave-57fn krb5kdc[24831](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321631, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:33:51.932682 26137 init.cc:377] Logged in from keytab as kudu/127.9.66.2@KRBTEST.COM (short username kudu)
I20260925 07:33:51.933100 26137 webserver.cc:460] Webserver started at http://127.9.66.2:34779/ using document root <none> and password file <none>
I20260925 07:33:51.933312 26137 fs_manager.cc:359] Metadata directory not provided
I20260925 07:33:51.933387 26137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:33:51.934693 26137 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:33:51.935310 26153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:33:51.935494 26137 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 07:33:51.935590 26137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-1/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-1/wal
uuid: "a3e2cf0a23c14e3e876406afcfdd6e07"
format_stamp: "Formatted at 2026-09-25 07:33:10 on dist-test-slave-57fn"
server_key: "249ac6722ef820f6a1335e766d83a839"
server_key_iv: "42dc75f8fcd51bab6b2ab4736611a7c7"
server_key_version: "encryptionkey@0"
I20260925 07:33:51.935906 26137 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:33:51.950861 26137 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:33:51.952555 26137 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:33:51.952718 26137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:33:51.952950 26137 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:33:51.953413 26160 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 07:33:51.954780 26137 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 07:33:51.954869 26137 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260925 07:33:51.954918 26137 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 07:33:51.955550 26137 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 07:33:51.955607 26137 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260925 07:33:51.955665 26160 tablet_bootstrap.cc:492] T 2fccfeef92d04be990f1091f1a284f62 P a3e2cf0a23c14e3e876406afcfdd6e07: Bootstrap starting.
I20260925 07:33:51.962321 26137 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.2:33299
I20260925 07:33:51.962420 26267 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.2:33299 every 8 connection(s)
I20260925 07:33:51.962705 26137 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-1/data/info.pb
I20260925 07:33:51.962965 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 26137
I20260925 07:33:51.963253 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.66.3:46457
--local_ip_for_outbound_sockets=127.9.66.3
--tserver_master_addrs=127.9.66.62:35239
--webserver_port=40087
--webserver_interface=127.9.66.3
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.3.keytab
--principal=kudu/127.9.66.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:43203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 07:33:51.963972 26270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:33:51.963411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:35239 (local address 127.9.66.2:46277)
0925 07:33:51.963680 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0925 07:33:51.963772 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:35239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":82,"threads_started":1}
W20260925 07:33:51.964255 26268 heartbeater.cc:644] Failed to heartbeat to 127.9.66.62:35239 (0 consecutive failures): Network error: Failed to ping master at 127.9.66.62:35239: Client connection negotiation failed: client connection to 127.9.66.62:35239: connect: Connection refused (error 111)
I20260925 07:33:51.964612 26270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:33:51.964475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:35239 (local address 127.9.66.2:60379)
0925 07:33:51.964556 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0925 07:33:51.964587 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:35239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260925 07:33:51.965011 26270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:33:51.964902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:35239 (local address 127.9.66.2:59199)
0925 07:33:51.964949 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0925 07:33:51.964976 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:35239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260925 07:33:51.965206 26268 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 07:33:51.966405 26160 log.cc:824] T 2fccfeef92d04be990f1091f1a284f62 P a3e2cf0a23c14e3e876406afcfdd6e07: Log is configured to *not* fsync() on all Append() calls
I20260925 07:33:51.968369 26160 tablet_bootstrap.cc:492] T 2fccfeef92d04be990f1091f1a284f62 P a3e2cf0a23c14e3e876406afcfdd6e07: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 07:33:51.968633 26160 tablet_bootstrap.cc:492] T 2fccfeef92d04be990f1091f1a284f62 P a3e2cf0a23c14e3e876406afcfdd6e07: Bootstrap complete.
I20260925 07:33:51.969583 26160 ts_tablet_manager.cc:1397] T 2fccfeef92d04be990f1091f1a284f62 P a3e2cf0a23c14e3e876406afcfdd6e07: Time spent bootstrapping tablet: real 0.014s user 0.011s sys 0.003s
I20260925 07:33:51.970724 26160 raft_consensus.cc:348] T 2fccfeef92d04be990f1091f1a284f62 P a3e2cf0a23c14e3e876406afcfdd6e07 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e2cf0a23c14e3e876406afcfdd6e07" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 33299 } } peers { permanent_uuid: "693a2a3014a34660ad9c23b17abceed7" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 46457 } } peers { permanent_uuid: "fb10411db4a0426fa59c5f405fbcaf03" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 34439 } }
I20260925 07:33:51.970966 26160 raft_consensus.cc:729] T 2fccfeef92d04be990f1091f1a284f62 P a3e2cf0a23c14e3e876406afcfdd6e07 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3e2cf0a23c14e3e876406afcfdd6e07, State: Initialized, Role: FOLLOWER
I20260925 07:33:51.971094 26160 consensus_queue.cc:260] T 2fccfeef92d04be990f1091f1a284f62 P a3e2cf0a23c14e3e876406afcfdd6e07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e2cf0a23c14e3e876406afcfdd6e07" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 33299 } } peers { permanent_uuid: "693a2a3014a34660ad9c23b17abceed7" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 46457 } } peers { permanent_uuid: "fb10411db4a0426fa59c5f405fbcaf03" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 34439 } }
I20260925 07:33:51.971355 26160 ts_tablet_manager.cc:1428] T 2fccfeef92d04be990f1091f1a284f62 P a3e2cf0a23c14e3e876406afcfdd6e07: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260925 07:33:51.971562 26270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:33:51.971495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:35239 (local address 127.9.66.2:47005)
0925 07:33:51.971523 (+ 28us) negotiation.cc:107] Waiting for socket to connect
0925 07:33:51.971547 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:35239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
I20260925 07:33:51.971864 26270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:33:51.971813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:35239 (local address 127.9.66.2:47889)
0925 07:33:51.971834 (+ 21us) negotiation.cc:107] Waiting for socket to connect
0925 07:33:51.971852 (+ 18us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:35239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13}
W20260925 07:33:52.043304 26271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:33:52.043480 26271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:33:52.043502 26271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:33:52.044597 26271 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:33:52.044636 26271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:33:52.044662 26271 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.66.3
I20260925 07:33:52.046075 26271 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:43203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.3.keytab
--principal=kudu/127.9.66.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.66.3:46457
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.66.3
--webserver_port=40087
--tserver_master_addrs=127.9.66.62:35239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26271
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.66.3
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:33:52.046314 26271 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:33:52.046537 26271 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:33:52.051148 26278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:33:52.051184 26281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:33:52.051126 26279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:33:52.051198 26271 server_base.cc:1034] running on GCE node
I20260925 07:33:52.051465 26271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:33:52.051717 26271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:33:52.052868 26271 hybrid_clock.cc:648] HybridClock initialized: now 1790321632052854 us; error 43 us; skew 500 ppm
Sep 25 07:33:52 dist-test-slave-57fn krb5kdc[24831](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321632, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:33:52.055259 26271 init.cc:377] Logged in from keytab as kudu/127.9.66.3@KRBTEST.COM (short username kudu)
I20260925 07:33:52.055709 26271 webserver.cc:460] Webserver started at http://127.9.66.3:40087/ using document root <none> and password file <none>
I20260925 07:33:52.055939 26271 fs_manager.cc:359] Metadata directory not provided
I20260925 07:33:52.056010 26271 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:33:52.057260 26271 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:33:52.057860 26288 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:33:52.058022 26271 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 07:33:52.058122 26271 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-2/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-2/wal
uuid: "693a2a3014a34660ad9c23b17abceed7"
format_stamp: "Formatted at 2026-09-25 07:33:10 on dist-test-slave-57fn"
server_key: "40e264dce66644c9faa1b5b316241f02"
server_key_iv: "d0fb262afd73437e77767ec549298845"
server_key_version: "encryptionkey@0"
I20260925 07:33:52.058432 26271 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:33:52.070528 26271 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:33:52.072340 26271 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:33:52.072487 26271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:33:52.072686 26271 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:33:52.072958 26271 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:33:52.073015 26271 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:33:52.073053 26271 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:33:52.073079 26271 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:33:52.078878 26271 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.3:46457
I20260925 07:33:52.078979 26401 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.3:46457 every 8 connection(s)
I20260925 07:33:52.079272 26271 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790321392278483-9480-0/minicluster-data/ts-2/data/info.pb
I20260925 07:33:52.080288 26404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:33:52.079934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:35239 (local address 127.9.66.3:41819)
0925 07:33:52.080096 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 07:33:52.080164 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:35239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":79,"threads_started":1}
W20260925 07:33:52.080703 26402 heartbeater.cc:644] Failed to heartbeat to 127.9.66.62:35239 (0 consecutive failures): Network error: Failed to ping master at 127.9.66.62:35239: Client connection negotiation failed: client connection to 127.9.66.62:35239: connect: Connection refused (error 111)
I20260925 07:33:52.080976 26404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:33:52.080889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:35239 (local address 127.9.66.3:36265)
0925 07:33:52.080935 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0925 07:33:52.080961 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:35239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260925 07:33:52.081203 26404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:33:52.081155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:35239 (local address 127.9.66.3:49683)
0925 07:33:52.081175 (+ 20us) negotiation.cc:107] Waiting for socket to connect
0925 07:33:52.081192 (+ 17us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:35239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12}
W20260925 07:33:52.081300 26402 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 07:33:52.088603 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 26271
I20260925 07:33:52.904987 26406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:33:52.904755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:35239 (local address 127.9.66.1:41623)
0925 07:33:52.904887 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0925 07:33:52.904922 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:35239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":67,"threads_started":1}
I20260925 07:33:52.972381 26407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:33:52.972208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:35239 (local address 127.9.66.2:57779)
0925 07:33:52.972312 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0925 07:33:52.972343 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:35239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":55,"threads_started":1}
I20260925 07:33:52.991003 26408 raft_consensus.cc:482] T 2fccfeef92d04be990f1091f1a284f62 P fb10411db4a0426fa59c5f405fbcaf03 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 07:33:52.991153 26408 raft_consensus.cc:504] T 2fccfeef92d04be990f1091f1a284f62 P fb10411db4a0426fa59c5f405fbcaf03 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e2cf0a23c14e3e876406afcfdd6e07" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 33299 } } peers { permanent_uuid: "693a2a3014a34660ad9c23b17abceed7" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 46457 } } peers { permanent_uuid: "fb10411db4a0426fa59c5f405fbcaf03" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 34439 } }
I20260925 07:33:52.991504 26408 leader_election.cc:288] T 2fccfeef92d04be990f1091f1a284f62 P fb10411db4a0426fa59c5f405fbcaf03 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a3e2cf0a23c14e3e876406afcfdd6e07 (127.9.66.2:33299), 693a2a3014a34660ad9c23b17abceed7 (127.9.66.3:46457)
Sep 25 07:33:52 dist-test-slave-57fn krb5kdc[24831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321631, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.1@KRBTEST.COM for kudu/127.9.66.3@KRBTEST.COM
Sep 25 07:33:52 dist-test-slave-57fn krb5kdc[24831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321631, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.1@KRBTEST.COM for kudu/127.9.66.2@KRBTEST.COM
I20260925 07:33:53.001667 26411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:33:52.991953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.1:33021 (local address 127.9.66.3:46457)
0925 07:33:52.992062 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0925 07:33:52.992073 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 07:33:52.992095 (+ 22us) server_negotiation.cc:407] Connection header received
0925 07:33:52.992498 (+ 403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:33:52.992499 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:33:52.992544 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:33:52.992672 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:33:52.993134 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:33:52.994067 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:33:52.995038 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:33:52.995415 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:33:52.995539 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:33:52.998015 (+ 2476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:33:52.998018 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:33:52.998019 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:33:52.998027 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:33:52.999916 (+ 1889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:33:53.000420 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:33:53.000424 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:33:53.000424 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:33:53.000517 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:33:53.000781 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:33:53.000782 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:33:53.000782 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:33:53.001015 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:33:53.001101 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 07:33:53.001407 (+ 306us) server_negotiation.cc:299] Negotiation successful
0925 07:33:53.001550 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75,"thread_start_us":46,"threads_started":1}
I20260925 07:33:53.001655 26406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:33:52.991708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.2:33299 (local address 127.9.66.1:44825)
0925 07:33:52.991763 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0925 07:33:52.991775 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 07:33:52.991848 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:33:52.992416 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:33:52.992420 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:33:52.992779 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:33:52.993018 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:33:52.993026 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:33:52.994163 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:33:52.994164 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:33:52.994935 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:33:52.994937 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:33:52.995519 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:33:52.995520 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:33:52.995624 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:33:52.996265 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:33:52.996269 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:33:52.998280 (+ 2011us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:33:53.000127 (+ 1847us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:33:53.000128 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:33:53.000128 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:33:53.000330 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:33:53.000589 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:33:53.000591 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:33:53.000591 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:33:53.000719 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:33:53.001120 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:33:53.001121 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:33:53.001235 (+ 114us) client_negotiation.cc:769] Sending connection context
0925 07:33:53.001344 (+ 109us) client_negotiation.cc:240] Negotiation successful
0925 07:33:53.001544 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":72}
I20260925 07:33:53.001679 26410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:33:52.991808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.1:44825 (local address 127.9.66.2:33299)
0925 07:33:52.992021 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0925 07:33:52.992032 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 07:33:52.992058 (+ 26us) server_negotiation.cc:407] Connection header received
0925 07:33:52.992174 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:33:52.992175 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:33:52.992214 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:33:52.992339 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:33:52.993122 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:33:52.994059 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:33:52.995040 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:33:52.995425 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:33:52.995549 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:33:52.998391 (+ 2842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:33:52.998394 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:33:52.998395 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:33:52.998406 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:33:53.000038 (+ 1632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:33:53.000430 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:33:53.000433 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:33:53.000433 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:33:53.000518 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:33:53.000805 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:33:53.000806 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:33:53.000806 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:33:53.001024 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:33:53.001120 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 07:33:53.001397 (+ 277us) server_negotiation.cc:299] Negotiation successful
0925 07:33:53.001563 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":99,"threads_started":1}
I20260925 07:33:53.001747 26409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:33:52.991843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.3:46457 (local address 127.9.66.1:33021)
0925 07:33:52.991968 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0925 07:33:52.991979 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 07:33:52.992041 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:33:52.992665 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:33:52.992666 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:33:52.992878 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:33:52.993016 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:33:52.993022 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:33:52.994169 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:33:52.994171 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:33:52.994936 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:33:52.994938 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:33:52.995515 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:33:52.995516 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:33:52.995630 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:33:52.996199 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:33:52.996207 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:33:52.997889 (+ 1682us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:33:53.000007 (+ 2118us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:33:53.000014 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:33:53.000017 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:33:53.000313 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:33:53.000591 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:33:53.000592 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:33:53.000593 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:33:53.000715 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:33:53.001087 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:33:53.001093 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:33:53.001234 (+ 141us) client_negotiation.cc:769] Sending connection context
0925 07:33:53.001345 (+ 111us) client_negotiation.cc:240] Negotiation successful
0925 07:33:53.001611 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"mutex_wait_us":137,"thread_start_us":69,"threads_started":1}
I20260925 07:33:53.002089 26222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fccfeef92d04be990f1091f1a284f62" candidate_uuid: "fb10411db4a0426fa59c5f405fbcaf03" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "a3e2cf0a23c14e3e876406afcfdd6e07" is_pre_election: true
I20260925 07:33:53.002264 26222 raft_consensus.cc:2454] T 2fccfeef92d04be990f1091f1a284f62 P a3e2cf0a23c14e3e876406afcfdd6e07 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb10411db4a0426fa59c5f405fbcaf03 in term 1.
I20260925 07:33:53.002568 26020 leader_election.cc:302] T 2fccfeef92d04be990f1091f1a284f62 P fb10411db4a0426fa59c5f405fbcaf03 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a3e2cf0a23c14e3e876406afcfdd6e07, fb10411db4a0426fa59c5f405fbcaf03; no voters:
I20260925 07:33:53.002696 26408 raft_consensus.cc:2781] T 2fccfeef92d04be990f1091f1a284f62 P fb10411db4a0426fa59c5f405fbcaf03 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 07:33:53.002758 26408 raft_consensus.cc:482] T 2fccfeef92d04be990f1091f1a284f62 P fb10411db4a0426fa59c5f405fbcaf03 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260925 07:33:53.002740 26020 leader_election.cc:341] T 2fccfeef92d04be990f1091f1a284f62 P fb10411db4a0426fa59c5f405fbcaf03 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 693a2a3014a34660ad9c23b17abceed7 (127.9.66.3:46457): Not found: Tablet not found: 2fccfeef92d04be990f1091f1a284f62
I20260925 07:33:53.002784 26408 raft_consensus.cc:3037] T 2fccfeef92d04be990f1091f1a284f62 P fb10411db4a0426fa59c5f405fbcaf03 [term 1 FOLLOWER]: Advancing to term 2
I20260925 07:33:53.004155 26408 raft_consensus.cc:504] T 2fccfeef92d04be990f1091f1a284f62 P fb10411db4a0426fa59c5f405fbcaf03 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e2cf0a23c14e3e876406afcfdd6e07" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 33299 } } peers { permanent_uuid: "693a2a3014a34660ad9c23b17abceed7" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 46457 } } peers { permanent_uuid: "fb10411db4a0426fa59c5f405fbcaf03" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 34439 } }
I20260925 07:33:53.004319 26408 leader_election.cc:288] T 2fccfeef92d04be990f1091f1a284f62 P fb10411db4a0426fa59c5f405fbcaf03 [CANDIDATE]: Term 2 election: Requested vote from peers a3e2cf0a23c14e3e876406afcfdd6e07 (127.9.66.2:33299), 693a2a3014a34660ad9c23b17abceed7 (127.9.66.3:46457)
I20260925 07:33:53.004519 26222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fccfeef92d04be990f1091f1a284f62" candidate_uuid: "fb10411db4a0426fa59c5f405fbcaf03" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "a3e2cf0a23c14e3e876406afcfdd6e07"
I20260925 07:33:53.004631 26222 raft_consensus.cc:3037] T 2fccfeef92d04be990f1091f1a284f62 P a3e2cf0a23c14e3e876406afcfdd6e07 [term 1 FOLLOWER]: Advancing to term 2
W20260925 07:33:53.004658 26020 leader_election.cc:341] T 2fccfeef92d04be990f1091f1a284f62 P fb10411db4a0426fa59c5f405fbcaf03 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 693a2a3014a34660ad9c23b17abceed7 (127.9.66.3:46457): Not found: Tablet not found: 2fccfeef92d04be990f1091f1a284f62
I20260925 07:33:53.005939 26222 raft_consensus.cc:2454] T 2fccfeef92d04be990f1091f1a284f62 P a3e2cf0a23c14e3e876406afcfdd6e07 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb10411db4a0426fa59c5f405fbcaf03 in term 2.
I20260925 07:33:53.006135 26020 leader_election.cc:302] T 2fccfeef92d04be990f1091f1a284f62 P fb10411db4a0426fa59c5f405fbcaf03 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a3e2cf0a23c14e3e876406afcfdd6e07, fb10411db4a0426fa59c5f405fbcaf03; no voters: 693a2a3014a34660ad9c23b17abceed7
I20260925 07:33:53.006244 26408 raft_consensus.cc:2781] T 2fccfeef92d04be990f1091f1a284f62 P fb10411db4a0426fa59c5f405fbcaf03 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 07:33:53.006361 26408 raft_consensus.cc:686] T 2fccfeef92d04be990f1091f1a284f62 P fb10411db4a0426fa59c5f405fbcaf03 [term 2 LEADER]: Becoming Leader. State: Replica: fb10411db4a0426fa59c5f405fbcaf03, State: Running, Role: LEADER
I20260925 07:33:53.006466 26408 consensus_queue.cc:237] T 2fccfeef92d04be990f1091f1a284f62 P fb10411db4a0426fa59c5f405fbcaf03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e2cf0a23c14e3e876406afcfdd6e07" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 33299 } } peers { permanent_uuid: "693a2a3014a34660ad9c23b17abceed7" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 46457 } } peers { permanent_uuid: "fb10411db4a0426fa59c5f405fbcaf03" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 34439 } }
I20260925 07:33:53.006700 26406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:33:53.006600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:35239 (local address 127.9.66.1:56315)
0925 07:33:53.006646 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0925 07:33:53.006673 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:35239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260925 07:33:53.081893 26414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:33:53.081698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:35239 (local address 127.9.66.3:40837)
0925 07:33:53.081820 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0925 07:33:53.081861 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:35239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":69,"threads_started":1}
W20260925 07:33:53.447753 26020 consensus_peers.cc:487] T 2fccfeef92d04be990f1091f1a284f62 P fb10411db4a0426fa59c5f405fbcaf03 -> Peer 693a2a3014a34660ad9c23b17abceed7 (127.9.66.3:46457): Couldn't send request to peer 693a2a3014a34660ad9c23b17abceed7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2fccfeef92d04be990f1091f1a284f62. This is attempt 1: this message will repeat every 5th retry.
I20260925 07:33:53.598712 26222 raft_consensus.cc:1261] T 2fccfeef92d04be990f1091f1a284f62 P a3e2cf0a23c14e3e876406afcfdd6e07 [term 2 FOLLOWER]: Refusing update from remote peer fb10411db4a0426fa59c5f405fbcaf03: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260925 07:33:53.599107 26408 consensus_queue.cc:1035] T 2fccfeef92d04be990f1091f1a284f62 P fb10411db4a0426fa59c5f405fbcaf03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3e2cf0a23c14e3e876406afcfdd6e07" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 33299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260925 07:33:53.600052 26417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:33:53.599866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:35239 (local address 127.9.66.2:44409)
0925 07:33:53.599968 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0925 07:33:53.600004 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:35239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":60,"threads_started":1}
I20260925 07:33:53.600953 26421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:33:53.600746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:35239 (local address 127.9.66.1:44391)
0925 07:33:53.600871 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0925 07:33:53.600906 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:35239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":69,"threads_started":1}
I20260925 07:33:53.601126 26417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:33:53.600964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:35239 (local address 127.9.66.2:47103)
0925 07:33:53.601031 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0925 07:33:53.601079 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:35239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260925 07:33:53.930086 26422 ts_tablet_manager.cc:927] T 2fccfeef92d04be990f1091f1a284f62 P 693a2a3014a34660ad9c23b17abceed7: Initiating tablet copy from peer fb10411db4a0426fa59c5f405fbcaf03 (127.9.66.1:34439)
I20260925 07:33:53.930277 26422 tablet_copy_client.cc:316] T 2fccfeef92d04be990f1091f1a284f62 P 693a2a3014a34660ad9c23b17abceed7: tablet copy: Beginning tablet copy session from remote peer at address 127.9.66.1:34439
Sep 25 07:33:53 dist-test-slave-57fn krb5kdc[24831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321632, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.3@KRBTEST.COM for kudu/127.9.66.1@KRBTEST.COM
I20260925 07:33:53.937832 26424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:33:53.930545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.1:34439 (local address 127.9.66.3:57485)
0925 07:33:53.930712 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 07:33:53.930731 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 07:33:53.930805 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:33:53.930980 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:33:53.930986 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:33:53.931291 (+ 305us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:33:53.931359 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:33:53.931362 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:33:53.932218 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:33:53.932219 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:33:53.932805 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:33:53.932808 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:33:53.933283 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:33:53.933284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:33:53.933375 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:33:53.933861 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:33:53.933865 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:33:53.935164 (+ 1299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:33:53.936830 (+ 1666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:33:53.936836 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:33:53.936837 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:33:53.937022 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:33:53.937218 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:33:53.937218 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:33:53.937219 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:33:53.937270 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:33:53.937543 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:33:53.937549 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:33:53.937602 (+ 53us) client_negotiation.cc:769] Sending connection context
0925 07:33:53.937669 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 07:33:53.937759 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":120,"threads_started":1}
I20260925 07:33:53.937925 26425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:33:53.930641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.3:57485 (local address 127.9.66.1:34439)
0925 07:33:53.930807 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 07:33:53.930813 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 07:33:53.930842 (+ 29us) server_negotiation.cc:407] Connection header received
0925 07:33:53.930870 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:33:53.930872 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:33:53.930890 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:33:53.930963 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:33:53.931454 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:33:53.932134 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:33:53.932901 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:33:53.933197 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:33:53.933283 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:33:53.935253 (+ 1970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:33:53.935261 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:33:53.935262 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:33:53.935269 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:33:53.936762 (+ 1493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:33:53.937090 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:33:53.937092 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:33:53.937092 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:33:53.937147 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:33:53.937317 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:33:53.937318 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:33:53.937318 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:33:53.937455 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:33:53.937550 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 07:33:53.937747 (+ 197us) server_negotiation.cc:299] Negotiation successful
0925 07:33:53.937814 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":83,"threads_started":1}
I20260925 07:33:53.938292 26107 tablet_copy_service.cc:140] P fb10411db4a0426fa59c5f405fbcaf03: Received BeginTabletCopySession request for tablet 2fccfeef92d04be990f1091f1a284f62 from peer 693a2a3014a34660ad9c23b17abceed7 ({username='kudu', principal='kudu/127.9.66.3@KRBTEST.COM'} at 127.9.66.3:57485)
I20260925 07:33:53.938354 26107 tablet_copy_service.cc:161] P fb10411db4a0426fa59c5f405fbcaf03: Beginning new tablet copy session on tablet 2fccfeef92d04be990f1091f1a284f62 from peer 693a2a3014a34660ad9c23b17abceed7 at {username='kudu', principal='kudu/127.9.66.3@KRBTEST.COM'} at 127.9.66.3:57485: session id = 693a2a3014a34660ad9c23b17abceed7-2fccfeef92d04be990f1091f1a284f62
I20260925 07:33:53.938938 26107 tablet_copy_source_session.cc:205] T 2fccfeef92d04be990f1091f1a284f62 P fb10411db4a0426fa59c5f405fbcaf03: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 07:33:53.939911 26422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fccfeef92d04be990f1091f1a284f62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:33:53.942333 26422 tablet_copy_client.cc:791] T 2fccfeef92d04be990f1091f1a284f62 P 693a2a3014a34660ad9c23b17abceed7: tablet copy: Starting download of 0 data blocks...
I20260925 07:33:53.942474 26422 tablet_copy_client.cc:655] T 2fccfeef92d04be990f1091f1a284f62 P 693a2a3014a34660ad9c23b17abceed7: tablet copy: Starting download of 1 WAL segments...
I20260925 07:33:53.943215 26422 tablet_copy_client.cc:531] T 2fccfeef92d04be990f1091f1a284f62 P 693a2a3014a34660ad9c23b17abceed7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 07:33:53.943866 26422 tablet_bootstrap.cc:492] T 2fccfeef92d04be990f1091f1a284f62 P 693a2a3014a34660ad9c23b17abceed7: Bootstrap starting.
I20260925 07:33:53.945151 26422 log.cc:824] T 2fccfeef92d04be990f1091f1a284f62 P 693a2a3014a34660ad9c23b17abceed7: Log is configured to *not* fsync() on all Append() calls
I20260925 07:33:53.947608 26422 tablet_bootstrap.cc:492] T 2fccfeef92d04be990f1091f1a284f62 P 693a2a3014a34660ad9c23b17abceed7: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 07:33:53.947831 26422 tablet_bootstrap.cc:492] T 2fccfeef92d04be990f1091f1a284f62 P 693a2a3014a34660ad9c23b17abceed7: Bootstrap complete.
I20260925 07:33:53.947961 26422 ts_tablet_manager.cc:1397] T 2fccfeef92d04be990f1091f1a284f62 P 693a2a3014a34660ad9c23b17abceed7: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260925 07:33:53.948876 26422 raft_consensus.cc:348] T 2fccfeef92d04be990f1091f1a284f62 P 693a2a3014a34660ad9c23b17abceed7 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e2cf0a23c14e3e876406afcfdd6e07" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 33299 } } peers { permanent_uuid: "693a2a3014a34660ad9c23b17abceed7" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 46457 } } peers { permanent_uuid: "fb10411db4a0426fa59c5f405fbcaf03" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 34439 } }
I20260925 07:33:53.949015 26422 raft_consensus.cc:729] T 2fccfeef92d04be990f1091f1a284f62 P 693a2a3014a34660ad9c23b17abceed7 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 693a2a3014a34660ad9c23b17abceed7, State: Initialized, Role: FOLLOWER
I20260925 07:33:53.949162 26422 consensus_queue.cc:260] T 2fccfeef92d04be990f1091f1a284f62 P 693a2a3014a34660ad9c23b17abceed7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e2cf0a23c14e3e876406afcfdd6e07" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 33299 } } peers { permanent_uuid: "693a2a3014a34660ad9c23b17abceed7" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 46457 } } peers { permanent_uuid: "fb10411db4a0426fa59c5f405fbcaf03" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 34439 } }
I20260925 07:33:53.949385 26422 ts_tablet_manager.cc:1428] T 2fccfeef92d04be990f1091f1a284f62 P 693a2a3014a34660ad9c23b17abceed7: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20260925 07:33:53.949661 26424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:33:53.949570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:35239 (local address 127.9.66.3:43973)
0925 07:33:53.949618 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0925 07:33:53.949642 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:35239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260925 07:33:53.949719 26107 tablet_copy_service.cc:342] P fb10411db4a0426fa59c5f405fbcaf03: Request end of tablet copy session 693a2a3014a34660ad9c23b17abceed7-2fccfeef92d04be990f1091f1a284f62 received from {username='kudu', principal='kudu/127.9.66.3@KRBTEST.COM'} at 127.9.66.3:57485
I20260925 07:33:53.949769 26107 tablet_copy_service.cc:433] P fb10411db4a0426fa59c5f405fbcaf03: ending tablet copy session 693a2a3014a34660ad9c23b17abceed7-2fccfeef92d04be990f1091f1a284f62 on tablet 2fccfeef92d04be990f1091f1a284f62 with peer 693a2a3014a34660ad9c23b17abceed7
I20260925 07:33:53.950042 26424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:33:53.949898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:35239 (local address 127.9.66.3:56689)
0925 07:33:53.949991 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0925 07:33:53.950020 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:35239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260925 07:33:54.391731 26429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:33:54.391554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:35239 (local address 127.9.66.1:39131)
0925 07:33:54.391652 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0925 07:33:54.391687 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:35239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":56,"threads_started":1}
I20260925 07:33:54.391695 26356 raft_consensus.cc:1203] T 2fccfeef92d04be990f1091f1a284f62 P 693a2a3014a34660ad9c23b17abceed7 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[2.3-2.3] Dedup: 2.3->[]
I20260925 07:33:54.392228 26424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:33:54.392105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:35239 (local address 127.9.66.3:60249)
0925 07:33:54.392159 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0925 07:33:54.392196 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:35239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260925 07:33:54.601764 26430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:33:54.601579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:35239 (local address 127.9.66.2:35987)
0925 07:33:54.601684 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0925 07:33:54.601720 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:35239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":49,"threads_started":1}
I20260925 07:33:55.392534 26431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:33:55.392300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:35239 (local address 127.9.66.1:45461)
0925 07:33:55.392417 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0925 07:33:55.392475 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:35239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":62,"threads_started":1}
I20260925 07:33:55.392795 26432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:33:55.392606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:35239 (local address 127.9.66.3:54927)
0925 07:33:55.392711 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0925 07:33:55.392745 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:35239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":66,"threads_started":1}
I20260925 07:33:55.602385 26433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:33:55.602200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:35239 (local address 127.9.66.2:35521)
0925 07:33:55.602307 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0925 07:33:55.602339 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:35239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":56,"threads_started":1}
I20260925 07:33:56.393255 26434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:33:56.392933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:35239 (local address 127.9.66.1:54373)
0925 07:33:56.393109 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0925 07:33:56.393188 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:35239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":84,"threads_started":1}
I20260925 07:33:56.393452 26435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:33:56.393205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:35239 (local address 127.9.66.3:40129)
0925 07:33:56.393331 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0925 07:33:56.393359 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:35239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":61,"threads_started":1}
I20260925 07:33:56.602989 26436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:33:56.602778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:35239 (local address 127.9.66.2:46505)
0925 07:33:56.602891 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0925 07:33:56.602932 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:35239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":59,"threads_started":1}
I20260925 07:33:57.394054 26437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:33:57.393799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:35239 (local address 127.9.66.1:43813)
0925 07:33:57.393949 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 07:33:57.393990 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:35239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":72,"threads_started":1}
I20260925 07:33:57.394129 26438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:33:57.393936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:35239 (local address 127.9.66.3:46735)
0925 07:33:57.394047 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0925 07:33:57.394078 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:35239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":60,"threads_started":1}
I20260925 07:33:57.394462 26437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:33:57.394384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:35239 (local address 127.9.66.1:38353)
0925 07:33:57.394424 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0925 07:33:57.394448 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:35239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260925 07:33:57.603710 26439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:33:57.603480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:35239 (local address 127.9.66.2:51147)
0925 07:33:57.603630 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 07:33:57.603666 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:35239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":72,"threads_started":1}
I20260925 07:33:58.394770 26440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:33:58.394526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:35239 (local address 127.9.66.3:33793)
0925 07:33:58.394656 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 07:33:58.394691 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:35239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":59,"threads_started":1}
I20260925 07:33:58.395045 26441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:33:58.394841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:35239 (local address 127.9.66.1:41247)
0925 07:33:58.394969 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0925 07:33:58.395007 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:35239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":64,"threads_started":1}
I20260925 07:33:58.604395 26442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:33:58.604185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:35239 (local address 127.9.66.2:43267)
0925 07:33:58.604294 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0925 07:33:58.604329 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:35239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":57,"threads_started":1}
I20260925 07:33:59.395709 26443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:33:59.395471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:35239 (local address 127.9.66.1:35233)
0925 07:33:59.395603 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0925 07:33:59.395644 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:35239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":75,"threads_started":1}
I20260925 07:33:59.397919 26444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:33:59.397702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:35239 (local address 127.9.66.3:44993)
0925 07:33:59.397816 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0925 07:33:59.397858 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:35239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":55,"threads_started":1}
I20260925 07:33:59.605120 26445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:33:59.604893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:35239 (local address 127.9.66.2:33323)
0925 07:33:59.605040 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 07:33:59.605076 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:35239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":62,"threads_started":1}
I20260925 07:34:00.396337 26447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:00.396116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:35239 (local address 127.9.66.1:59035)
0925 07:34:00.396253 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 07:34:00.396289 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:35239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":61,"threads_started":1}
I20260925 07:34:00.398583 26448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:00.398368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:35239 (local address 127.9.66.3:47177)
0925 07:34:00.398508 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 07:34:00.398544 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:35239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":68,"threads_started":1}
I20260925 07:34:00.605878 26449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:00.605643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:35239 (local address 127.9.66.2:54023)
0925 07:34:00.605788 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 07:34:00.605826 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:35239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":76,"threads_started":1}
I20260925 07:34:01.397056 26450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:01.396831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:35239 (local address 127.9.66.1:43479)
0925 07:34:01.396956 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0925 07:34:01.396991 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:35239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":61,"threads_started":1}
I20260925 07:34:01.399271 26451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:01.399077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:35239 (local address 127.9.66.3:50353)
0925 07:34:01.399201 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0925 07:34:01.399234 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:35239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":76,"threads_started":1}
I20260925 07:34:01.606549 26452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:01.606332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:35239 (local address 127.9.66.2:47617)
0925 07:34:01.606469 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 07:34:01.606503 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:35239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":80,"threads_started":1}
I20260925 07:34:02.397899 26453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:02.397670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:35239 (local address 127.9.66.1:49749)
0925 07:34:02.397801 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0925 07:34:02.397840 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:35239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":64,"threads_started":1}
I20260925 07:34:02.399863 26454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:02.399691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:35239 (local address 127.9.66.3:35891)
0925 07:34:02.399792 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0925 07:34:02.399826 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:35239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":56,"threads_started":1}
I20260925 07:34:02.607234 26455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:02.606982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:35239 (local address 127.9.66.2:32859)
0925 07:34:02.607153 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 07:34:02.607189 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:35239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":116,"threads_started":1}
I20260925 07:34:03.398615 26456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:03.398401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:35239 (local address 127.9.66.1:51181)
0925 07:34:03.398534 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0925 07:34:03.398571 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:35239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":61,"threads_started":1}
I20260925 07:34:03.400529 26457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:03.400340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:35239 (local address 127.9.66.3:38139)
0925 07:34:03.400462 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0925 07:34:03.400491 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:35239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":75,"threads_started":1}
I20260925 07:34:03.607932 26458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:03.607700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:35239 (local address 127.9.66.2:51753)
0925 07:34:03.607830 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 07:34:03.607881 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:35239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":59,"threads_started":1}
I20260925 07:34:04.399387 26459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:04.399103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:35239 (local address 127.9.66.1:54251)
0925 07:34:04.399270 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 07:34:04.399309 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:35239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":81,"threads_started":1}
I20260925 07:34:04.401168 26460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:04.400992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:35239 (local address 127.9.66.3:43951)
0925 07:34:04.401100 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0925 07:34:04.401132 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:35239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":54,"threads_started":1}
I20260925 07:34:04.608562 26461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:04.608364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:35239 (local address 127.9.66.2:38205)
0925 07:34:04.608488 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0925 07:34:04.608518 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:35239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":68,"threads_started":1}
I20260925 07:34:05.400084 26462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:05.399886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:35239 (local address 127.9.66.1:45839)
0925 07:34:05.400006 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0925 07:34:05.400039 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:35239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":62,"threads_started":1}
I20260925 07:34:05.401798 26463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:05.401617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:35239 (local address 127.9.66.3:56869)
0925 07:34:05.401723 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0925 07:34:05.401748 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:35239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83,"thread_start_us":61,"threads_started":1}
I20260925 07:34:05.609305 26464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:05.609036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:35239 (local address 127.9.66.2:49011)
0925 07:34:05.609227 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0925 07:34:05.609262 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:35239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":113,"threads_started":1}
I20260925 07:34:06.400779 26465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:06.400543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:35239 (local address 127.9.66.1:56565)
0925 07:34:06.400696 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 07:34:06.400731 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:35239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":66,"threads_started":1}
I20260925 07:34:06.402352 26466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:06.402160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:35239 (local address 127.9.66.3:37811)
0925 07:34:06.402280 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0925 07:34:06.402316 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:35239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":63,"threads_started":1}
I20260925 07:34:06.609941 26467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:06.609731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:35239 (local address 127.9.66.2:43369)
0925 07:34:06.609844 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0925 07:34:06.609891 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.62:35239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":60,"threads_started":1}
I20260925 07:34:07.098701 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 25970
I20260925 07:34:07.103999 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 26137
I20260925 07:34:07.108886 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 26271
2026-09-25T07:34:07Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58424 ms)
[----------] 30 tests from SecurityITest (254839 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790321392278483-9480-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 07:34:07 dist-test-slave-57fn krb5kdc[26470](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 07:34:07 dist-test-slave-57fn krb5kdc[26470](info): set up 2 sockets
Sep 25 07:34:07 dist-test-slave-57fn krb5kdc[26470](info): commencing operation
krb5kdc: starting...
W20260925 07:34:08.137156 9480 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790321392278483-9480-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790321392278483-9480-0/krb5kdc/test-user.keytab.
Sep 25 07:34:08 dist-test-slave-57fn krb5kdc[26470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321648, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T07:34:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T07:34:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.66.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.66.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.66.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab.
Entry for principal kudu/127.9.66.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab.
I20260925 07:34:08.256155 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790321392278483-9480-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790321392278483-9480-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790321392278483-9480-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.66.62:42049
--webserver_interface=127.9.66.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab
--principal=kudu/127.9.66.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:35697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.66.62:42049
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:34:08.332824 26483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:34:08.333070 26483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:34:08.333110 26483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:34:08.334245 26483 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:34:08.334316 26483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:34:08.334340 26483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:34:08.334368 26483 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:34:08.334399 26483 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:34:08.335830 26483 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:35697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790321392278483-9480-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790321392278483-9480-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.66.62:42049
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab
--principal=kudu/127.9.66.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.66.62:42049
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.66.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26483
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790321392278483-9480-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:34:08.336095 26483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:34:08.336315 26483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:34:08.341115 26488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:34:08.341243 26483 server_base.cc:1034] running on GCE node
W20260925 07:34:08.341113 26491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:34:08.341184 26489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:34:08.341642 26483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:34:08.341951 26483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:34:08.343112 26483 hybrid_clock.cc:648] HybridClock initialized: now 1790321648343109 us; error 34 us; skew 500 ppm
Sep 25 07:34:08 dist-test-slave-57fn krb5kdc[26470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321648, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:34:08.345577 26483 init.cc:377] Logged in from keytab as kudu/127.9.66.62@KRBTEST.COM (short username kudu)
I20260925 07:34:08.346012 26483 webserver.cc:460] Webserver started at http://127.9.66.62:38591/ using document root <none> and password file <none>
I20260925 07:34:08.346236 26483 fs_manager.cc:359] Metadata directory not provided
I20260925 07:34:08.346302 26483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:34:08.346447 26483 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:34:08.347437 26483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790321392278483-9480-0/minicluster-data/master-0/data/instance:
uuid: "d0adaabfb25a41eaadf4baf35e082c22"
format_stamp: "Formatted at 2026-09-25 07:34:08 on dist-test-slave-57fn"
server_key: "6280229ceb0b20abdff52c86e2d4348d"
server_key_iv: "d735c5856a20461c1c2c7e6386c107ee"
server_key_version: "encryptionkey@0"
I20260925 07:34:08.347783 26483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790321392278483-9480-0/minicluster-data/master-0/wal/instance:
uuid: "d0adaabfb25a41eaadf4baf35e082c22"
format_stamp: "Formatted at 2026-09-25 07:34:08 on dist-test-slave-57fn"
server_key: "6280229ceb0b20abdff52c86e2d4348d"
server_key_iv: "d735c5856a20461c1c2c7e6386c107ee"
server_key_version: "encryptionkey@0"
I20260925 07:34:08.349061 26483 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 07:34:08.349805 26498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:34:08.349959 26483 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 07:34:08.350075 26483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790321392278483-9480-0/minicluster-data/master-0/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790321392278483-9480-0/minicluster-data/master-0/wal
uuid: "d0adaabfb25a41eaadf4baf35e082c22"
format_stamp: "Formatted at 2026-09-25 07:34:08 on dist-test-slave-57fn"
server_key: "6280229ceb0b20abdff52c86e2d4348d"
server_key_iv: "d735c5856a20461c1c2c7e6386c107ee"
server_key_version: "encryptionkey@0"
I20260925 07:34:08.350145 26483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790321392278483-9480-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790321392278483-9480-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790321392278483-9480-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:34:08.376161 26483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:34:08.378099 26483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:34:08.378259 26483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:34:08.382320 26483 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.62:42049
I20260925 07:34:08.382382 26550 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.62:42049 every 8 connection(s)
I20260925 07:34:08.382732 26483 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
I20260925 07:34:08.383533 26551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:34:08.385687 26551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0adaabfb25a41eaadf4baf35e082c22: Bootstrap starting.
I20260925 07:34:08.386206 26551 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d0adaabfb25a41eaadf4baf35e082c22: Neither blocks nor log segments found. Creating new log.
I20260925 07:34:08.386430 26551 log.cc:824] T 00000000000000000000000000000000 P d0adaabfb25a41eaadf4baf35e082c22: Log is configured to *not* fsync() on all Append() calls
I20260925 07:34:08.387051 26551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0adaabfb25a41eaadf4baf35e082c22: No bootstrap required, opened a new log
I20260925 07:34:08.388718 26551 raft_consensus.cc:348] T 00000000000000000000000000000000 P d0adaabfb25a41eaadf4baf35e082c22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0adaabfb25a41eaadf4baf35e082c22" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 42049 } }
I20260925 07:34:08.388849 26551 raft_consensus.cc:374] T 00000000000000000000000000000000 P d0adaabfb25a41eaadf4baf35e082c22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:34:08.388880 26551 raft_consensus.cc:729] T 00000000000000000000000000000000 P d0adaabfb25a41eaadf4baf35e082c22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0adaabfb25a41eaadf4baf35e082c22, State: Initialized, Role: FOLLOWER
I20260925 07:34:08.388983 26551 consensus_queue.cc:260] T 00000000000000000000000000000000 P d0adaabfb25a41eaadf4baf35e082c22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0adaabfb25a41eaadf4baf35e082c22" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 42049 } }
I20260925 07:34:08.389034 26551 raft_consensus.cc:388] T 00000000000000000000000000000000 P d0adaabfb25a41eaadf4baf35e082c22 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 07:34:08.389055 26551 raft_consensus.cc:482] T 00000000000000000000000000000000 P d0adaabfb25a41eaadf4baf35e082c22 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 07:34:08.389119 26551 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d0adaabfb25a41eaadf4baf35e082c22 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:34:08.389689 26551 raft_consensus.cc:504] T 00000000000000000000000000000000 P d0adaabfb25a41eaadf4baf35e082c22 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0adaabfb25a41eaadf4baf35e082c22" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 42049 } }
I20260925 07:34:08.389797 26551 leader_election.cc:302] T 00000000000000000000000000000000 P d0adaabfb25a41eaadf4baf35e082c22 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d0adaabfb25a41eaadf4baf35e082c22; no voters:
I20260925 07:34:08.389972 26551 leader_election.cc:288] T 00000000000000000000000000000000 P d0adaabfb25a41eaadf4baf35e082c22 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 07:34:08.390051 26554 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d0adaabfb25a41eaadf4baf35e082c22 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:34:08.390223 26554 raft_consensus.cc:686] T 00000000000000000000000000000000 P d0adaabfb25a41eaadf4baf35e082c22 [term 1 LEADER]: Becoming Leader. State: Replica: d0adaabfb25a41eaadf4baf35e082c22, State: Running, Role: LEADER
I20260925 07:34:08.390244 26551 sys_catalog.cc:564] T 00000000000000000000000000000000 P d0adaabfb25a41eaadf4baf35e082c22 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:34:08.390305 26554 consensus_queue.cc:237] T 00000000000000000000000000000000 P d0adaabfb25a41eaadf4baf35e082c22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0adaabfb25a41eaadf4baf35e082c22" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 42049 } }
I20260925 07:34:08.390733 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 26483
I20260925 07:34:08.390870 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790321392278483-9480-0/minicluster-data/master-0/wal/instance
I20260925 07:34:08.391078 9480 external_mini_cluster.cc:1428] Setting key 48aa08b6c1210a81f5df06acc8fe1ea7
I20260925 07:34:08.391278 26555 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0adaabfb25a41eaadf4baf35e082c22 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d0adaabfb25a41eaadf4baf35e082c22" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0adaabfb25a41eaadf4baf35e082c22" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 42049 } } }
I20260925 07:34:08.391372 26555 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0adaabfb25a41eaadf4baf35e082c22 [sys.catalog]: This master's current role is: LEADER
I20260925 07:34:08.391556 26556 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0adaabfb25a41eaadf4baf35e082c22 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0adaabfb25a41eaadf4baf35e082c22. Latest consensus state: current_term: 1 leader_uuid: "d0adaabfb25a41eaadf4baf35e082c22" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0adaabfb25a41eaadf4baf35e082c22" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 42049 } } }
I20260925 07:34:08.391613 26556 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0adaabfb25a41eaadf4baf35e082c22 [sys.catalog]: This master's current role is: LEADER
I20260925 07:34:08.391736 26565 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 07:34:08.392201 26565 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 07:34:08.393785 26565 catalog_manager.cc:1347] Generated new cluster ID: 6ca435b1c4f64780a60ca9ae31180694
I20260925 07:34:08.393837 26565 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 07:34:08 dist-test-slave-57fn krb5kdc[26470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321648, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.66.62@KRBTEST.COM
I20260925 07:34:08.399370 26565 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 07:34:08.400005 26565 catalog_manager.cc:1504] Loading token signing keys...
I20260925 07:34:08.403846 26572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:08.391759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50988 (local address 127.9.66.62:42049)
0925 07:34:08.392099 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:08.392109 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:08.392141 (+ 32us) server_negotiation.cc:407] Connection header received
0925 07:34:08.392251 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:08.392258 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:08.392311 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:08.392446 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:34:08.393173 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:08.394363 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:08.395089 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:08.395533 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:08.395644 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:08.397871 (+ 2227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:34:08.397875 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:34:08.397875 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:34:08.397893 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:34:08.402289 (+ 4396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:34:08.402653 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:34:08.402657 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:34:08.402657 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:34:08.402755 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:34:08.402957 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:34:08.402958 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:34:08.402958 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:34:08.403165 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:34:08.403404 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:08.403496 (+ 92us) server_negotiation.cc:299] Negotiation successful
0925 07:34:08.403646 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":68,"threads_started":1}
I20260925 07:34:08.404598 26565 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d0adaabfb25a41eaadf4baf35e082c22: Generated new TSK 0
I20260925 07:34:08.404737 26565 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.66.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.66.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.66.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.1.keytab.
Entry for principal kudu/127.9.66.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.1.keytab.
I20260925 07:34:08.432413 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790321392278483-9480-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790321392278483-9480-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790321392278483-9480-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790321392278483-9480-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.66.1:0
--local_ip_for_outbound_sockets=127.9.66.1
--webserver_interface=127.9.66.1
--webserver_port=0
--tserver_master_addrs=127.9.66.62:42049
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.1.keytab
--principal=kudu/127.9.66.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:35697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:34:08.515585 26577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:34:08.515772 26577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:34:08.515795 26577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:34:08.517014 26577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:34:08.517055 26577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:34:08.517081 26577 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.66.1
I20260925 07:34:08.518538 26577 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:35697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790321392278483-9480-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790321392278483-9480-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.1.keytab
--principal=kudu/127.9.66.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.66.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790321392278483-9480-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.66.1
--webserver_port=0
--tserver_master_addrs=127.9.66.62:42049
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26577
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.66.1
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790321392278483-9480-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:34:08.518770 26577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:34:08.519018 26577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:34:08.524627 26585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:34:08.524765 26577 server_base.cc:1034] running on GCE node
W20260925 07:34:08.524794 26582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:34:08.524648 26583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:34:08.525113 26577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:34:08.525419 26577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:34:08.526580 26577 hybrid_clock.cc:648] HybridClock initialized: now 1790321648526567 us; error 29 us; skew 500 ppm
Sep 25 07:34:08 dist-test-slave-57fn krb5kdc[26470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321648, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:34:08.528945 26577 init.cc:377] Logged in from keytab as kudu/127.9.66.1@KRBTEST.COM (short username kudu)
I20260925 07:34:08.529336 26577 webserver.cc:460] Webserver started at http://127.9.66.1:38031/ using document root <none> and password file <none>
I20260925 07:34:08.529608 26577 fs_manager.cc:359] Metadata directory not provided
I20260925 07:34:08.529671 26577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:34:08.529807 26577 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:34:08.530731 26577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790321392278483-9480-0/minicluster-data/ts-0/data/instance:
uuid: "b87af5315b1648c28d1ee73d9a0ba874"
format_stamp: "Formatted at 2026-09-25 07:34:08 on dist-test-slave-57fn"
server_key: "9ba9c24a80cd1590518e60b6e086e380"
server_key_iv: "06dccbd2a0c5a0b6dedca74b6bee4fc7"
server_key_version: "encryptionkey@0"
I20260925 07:34:08.531097 26577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790321392278483-9480-0/minicluster-data/ts-0/wal/instance:
uuid: "b87af5315b1648c28d1ee73d9a0ba874"
format_stamp: "Formatted at 2026-09-25 07:34:08 on dist-test-slave-57fn"
server_key: "9ba9c24a80cd1590518e60b6e086e380"
server_key_iv: "06dccbd2a0c5a0b6dedca74b6bee4fc7"
server_key_version: "encryptionkey@0"
I20260925 07:34:08.532428 26577 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 07:34:08.533203 26592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:34:08.533389 26577 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 07:34:08.533483 26577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790321392278483-9480-0/minicluster-data/ts-0/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790321392278483-9480-0/minicluster-data/ts-0/wal
uuid: "b87af5315b1648c28d1ee73d9a0ba874"
format_stamp: "Formatted at 2026-09-25 07:34:08 on dist-test-slave-57fn"
server_key: "9ba9c24a80cd1590518e60b6e086e380"
server_key_iv: "06dccbd2a0c5a0b6dedca74b6bee4fc7"
server_key_version: "encryptionkey@0"
I20260925 07:34:08.533620 26577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790321392278483-9480-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790321392278483-9480-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790321392278483-9480-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:34:08.551251 26577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:34:08.553740 26577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:34:08.553897 26577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:34:08.554133 26577 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:34:08.554450 26577 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:34:08.554500 26577 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:34:08.554534 26577 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:34:08.554560 26577 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:34:08.560289 26577 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.1:37825
I20260925 07:34:08.560326 26705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.1:37825 every 8 connection(s)
I20260925 07:34:08.560659 26577 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790321392278483-9480-0/minicluster-data/ts-0/data/info.pb
Sep 25 07:34:08 dist-test-slave-57fn krb5kdc[26470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321648, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.1@KRBTEST.COM for kudu/127.9.66.62@KRBTEST.COM
I20260925 07:34:08.566922 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 26577
I20260925 07:34:08.567060 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790321392278483-9480-0/minicluster-data/ts-0/wal/instance
I20260925 07:34:08.567278 9480 external_mini_cluster.cc:1428] Setting key b183e860aae73fba7ba44a9ccaacc9aa
I20260925 07:34:08.569690 26572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:08.561392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.1:60649 (local address 127.9.66.62:42049)
0925 07:34:08.561462 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:08.561465 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:08.561689 (+ 224us) server_negotiation.cc:407] Connection header received
0925 07:34:08.562136 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:08.562137 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:08.562147 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:08.562197 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:34:08.563028 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:08.563507 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:08.564408 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:08.564584 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:08.564631 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:08.566719 (+ 2088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:34:08.566721 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:34:08.566721 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:34:08.566727 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:34:08.568310 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:34:08.568720 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:34:08.568721 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:34:08.568722 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:34:08.568782 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:34:08.569036 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:34:08.569037 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:34:08.569038 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:34:08.569148 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:34:08.569232 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:08.569496 (+ 264us) server_negotiation.cc:299] Negotiation successful
0925 07:34:08.569606 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 07:34:08.569845 26708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:08.561343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:42049 (local address 127.9.66.1:60649)
0925 07:34:08.561547 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0925 07:34:08.561596 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 07:34:08.562057 (+ 461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:34:08.562245 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:34:08.562248 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:34:08.562765 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:34:08.562948 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:08.562952 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:08.563584 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:08.563585 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:08.564325 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:08.564326 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:08.564651 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:08.564652 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:08.564743 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:08.565248 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:34:08.565255 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:34:08.566637 (+ 1382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:34:08.568431 (+ 1794us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:34:08.568438 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:34:08.568440 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:34:08.568646 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:34:08.568849 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:34:08.568850 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:34:08.568850 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:34:08.568969 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:34:08.569223 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:34:08.569229 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:34:08.569365 (+ 136us) client_negotiation.cc:769] Sending connection context
0925 07:34:08.569461 (+ 96us) client_negotiation.cc:240] Negotiation successful
0925 07:34:08.569628 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":103,"threads_started":1}
I20260925 07:34:08.570514 26706 heartbeater.cc:347] Connected to a master server at 127.9.66.62:42049
I20260925 07:34:08.570613 26706 heartbeater.cc:464] Registering TS with master...
I20260925 07:34:08.570773 26706 heartbeater.cc:511] Master 127.9.66.62:42049 requested a full tablet report, sending...
I20260925 07:34:08.571319 26515 ts_manager.cc:194] Registered new tserver with Master: b87af5315b1648c28d1ee73d9a0ba874 (127.9.66.1:37825)
I20260925 07:34:08.572244 26515 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.66.1@KRBTEST.COM'} at 127.9.66.1:60649
WARNING: no policy specified for kudu/127.9.66.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.66.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.66.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.2.keytab.
Entry for principal kudu/127.9.66.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.2.keytab.
I20260925 07:34:08.592715 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790321392278483-9480-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790321392278483-9480-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790321392278483-9480-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790321392278483-9480-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.66.2:0
--local_ip_for_outbound_sockets=127.9.66.2
--webserver_interface=127.9.66.2
--webserver_port=0
--tserver_master_addrs=127.9.66.62:42049
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.2.keytab
--principal=kudu/127.9.66.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:35697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:34:08.668776 26711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:34:08.668994 26711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:34:08.669042 26711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:34:08.670184 26711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:34:08.670249 26711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:34:08.670292 26711 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.66.2
I20260925 07:34:08.671710 26711 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:35697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790321392278483-9480-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790321392278483-9480-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.2.keytab
--principal=kudu/127.9.66.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.66.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790321392278483-9480-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.66.2
--webserver_port=0
--tserver_master_addrs=127.9.66.62:42049
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26711
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.66.2
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790321392278483-9480-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:34:08.671963 26711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:34:08.672178 26711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:34:08.676887 26719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:34:08.676877 26716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:34:08.676963 26711 server_base.cc:1034] running on GCE node
W20260925 07:34:08.676877 26717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:34:08.677289 26711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:34:08.677644 26711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:34:08.678803 26711 hybrid_clock.cc:648] HybridClock initialized: now 1790321648678779 us; error 41 us; skew 500 ppm
Sep 25 07:34:08 dist-test-slave-57fn krb5kdc[26470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321648, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:34:08.681208 26711 init.cc:377] Logged in from keytab as kudu/127.9.66.2@KRBTEST.COM (short username kudu)
I20260925 07:34:08.681701 26711 webserver.cc:460] Webserver started at http://127.9.66.2:38899/ using document root <none> and password file <none>
I20260925 07:34:08.681905 26711 fs_manager.cc:359] Metadata directory not provided
I20260925 07:34:08.681964 26711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:34:08.682111 26711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:34:08.683066 26711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790321392278483-9480-0/minicluster-data/ts-1/data/instance:
uuid: "d89b70fd747b44a5bef016c8379dd0e4"
format_stamp: "Formatted at 2026-09-25 07:34:08 on dist-test-slave-57fn"
server_key: "0967bbbbc261c06744c2000284921de6"
server_key_iv: "172c4ef40a2a14677493312030921b6c"
server_key_version: "encryptionkey@0"
I20260925 07:34:08.683406 26711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790321392278483-9480-0/minicluster-data/ts-1/wal/instance:
uuid: "d89b70fd747b44a5bef016c8379dd0e4"
format_stamp: "Formatted at 2026-09-25 07:34:08 on dist-test-slave-57fn"
server_key: "0967bbbbc261c06744c2000284921de6"
server_key_iv: "172c4ef40a2a14677493312030921b6c"
server_key_version: "encryptionkey@0"
I20260925 07:34:08.684578 26711 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 07:34:08.685225 26726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:34:08.685412 26711 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 07:34:08.685497 26711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790321392278483-9480-0/minicluster-data/ts-1/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790321392278483-9480-0/minicluster-data/ts-1/wal
uuid: "d89b70fd747b44a5bef016c8379dd0e4"
format_stamp: "Formatted at 2026-09-25 07:34:08 on dist-test-slave-57fn"
server_key: "0967bbbbc261c06744c2000284921de6"
server_key_iv: "172c4ef40a2a14677493312030921b6c"
server_key_version: "encryptionkey@0"
I20260925 07:34:08.685578 26711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790321392278483-9480-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790321392278483-9480-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790321392278483-9480-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:34:08.697592 26711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:34:08.699615 26711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:34:08.699719 26711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:34:08.699908 26711 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:34:08.700224 26711 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:34:08.700278 26711 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:34:08.700309 26711 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:34:08.700335 26711 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:34:08.705716 26711 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.2:41287
I20260925 07:34:08.705807 26839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.2:41287 every 8 connection(s)
I20260925 07:34:08.706071 26711 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790321392278483-9480-0/minicluster-data/ts-1/data/info.pb
I20260925 07:34:08.706779 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 26711
I20260925 07:34:08.706908 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790321392278483-9480-0/minicluster-data/ts-1/wal/instance
I20260925 07:34:08.707119 9480 external_mini_cluster.cc:1428] Setting key 234d9191e84bea4d6ee82a28aeb837cc
Sep 25 07:34:08 dist-test-slave-57fn krb5kdc[26470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321648, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.2@KRBTEST.COM for kudu/127.9.66.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.66.3@KRBTEST.COM; defaulting to no policy
I20260925 07:34:08.715617 26572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:08.706808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.2:55517 (local address 127.9.66.62:42049)
0925 07:34:08.706901 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:08.706903 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:08.707010 (+ 107us) server_negotiation.cc:407] Connection header received
0925 07:34:08.707602 (+ 592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:08.707603 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:08.707615 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:08.707657 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:34:08.708564 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:08.709073 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:08.710339 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:08.710523 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:08.710602 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:08.712843 (+ 2241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:34:08.712844 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:34:08.712844 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:34:08.712852 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:34:08.714218 (+ 1366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:34:08.714654 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:34:08.714655 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:34:08.714655 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:34:08.714713 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:34:08.714950 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:34:08.714951 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:34:08.714951 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:34:08.715092 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:34:08.715168 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:08.715418 (+ 250us) server_negotiation.cc:299] Negotiation successful
0925 07:34:08.715507 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 07:34:08.715797 26842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:08.706734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:42049 (local address 127.9.66.2:55517)
0925 07:34:08.706900 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 07:34:08.706936 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 07:34:08.707503 (+ 567us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:34:08.707717 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:34:08.707720 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:34:08.708280 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:34:08.708467 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:08.708471 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:08.709165 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:08.709166 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:08.710248 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:08.710251 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:08.710635 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:08.710636 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:08.710774 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:08.711225 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:34:08.711236 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:34:08.712755 (+ 1519us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:34:08.714320 (+ 1565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:34:08.714327 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:34:08.714329 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:34:08.714568 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:34:08.714783 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:34:08.714784 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:34:08.714784 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:34:08.714884 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:34:08.715159 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:34:08.715165 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:34:08.715296 (+ 131us) client_negotiation.cc:769] Sending connection context
0925 07:34:08.715398 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 07:34:08.715580 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":92,"threads_started":1}
I20260925 07:34:08.716271 26840 heartbeater.cc:347] Connected to a master server at 127.9.66.62:42049
I20260925 07:34:08.716364 26840 heartbeater.cc:464] Registering TS with master...
I20260925 07:34:08.716535 26840 heartbeater.cc:511] Master 127.9.66.62:42049 requested a full tablet report, sending...
I20260925 07:34:08.716892 26515 ts_manager.cc:194] Registered new tserver with Master: d89b70fd747b44a5bef016c8379dd0e4 (127.9.66.2:41287)
I20260925 07:34:08.717255 26515 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.66.2@KRBTEST.COM'} at 127.9.66.2:55517
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.66.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.66.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.3.keytab.
Entry for principal kudu/127.9.66.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.3.keytab.
I20260925 07:34:08.731964 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790321392278483-9480-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790321392278483-9480-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790321392278483-9480-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790321392278483-9480-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.66.3:0
--local_ip_for_outbound_sockets=127.9.66.3
--webserver_interface=127.9.66.3
--webserver_port=0
--tserver_master_addrs=127.9.66.62:42049
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.3.keytab
--principal=kudu/127.9.66.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:35697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:34:08.812337 26845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:34:08.812517 26845 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:34:08.812539 26845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:34:08.813689 26845 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:34:08.813755 26845 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:34:08.813781 26845 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.66.3
I20260925 07:34:08.815153 26845 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:35697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790321392278483-9480-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790321392278483-9480-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.3.keytab
--principal=kudu/127.9.66.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.66.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790321392278483-9480-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.66.3
--webserver_port=0
--tserver_master_addrs=127.9.66.62:42049
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26845
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.66.3
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790321392278483-9480-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:34:08.815373 26845 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:34:08.815608 26845 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:34:08.820473 26851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:34:08.820546 26845 server_base.cc:1034] running on GCE node
W20260925 07:34:08.820529 26853 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:34:08.820488 26850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:34:08.820858 26845 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:34:08.821038 26845 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:34:08.822163 26845 hybrid_clock.cc:648] HybridClock initialized: now 1790321648822154 us; error 25 us; skew 500 ppm
Sep 25 07:34:08 dist-test-slave-57fn krb5kdc[26470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321648, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:34:08.824461 26845 init.cc:377] Logged in from keytab as kudu/127.9.66.3@KRBTEST.COM (short username kudu)
I20260925 07:34:08.824797 26845 webserver.cc:460] Webserver started at http://127.9.66.3:33327/ using document root <none> and password file <none>
I20260925 07:34:08.824970 26845 fs_manager.cc:359] Metadata directory not provided
I20260925 07:34:08.825016 26845 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:34:08.825095 26845 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:34:08.826071 26845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790321392278483-9480-0/minicluster-data/ts-2/data/instance:
uuid: "51fb8e311c694370a5aa02d0e6411927"
format_stamp: "Formatted at 2026-09-25 07:34:08 on dist-test-slave-57fn"
server_key: "34ab6e288c1f35539d4ea06afc65a48c"
server_key_iv: "4fb79e72ca00484ad61393ade3aa0429"
server_key_version: "encryptionkey@0"
I20260925 07:34:08.826368 26845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790321392278483-9480-0/minicluster-data/ts-2/wal/instance:
uuid: "51fb8e311c694370a5aa02d0e6411927"
format_stamp: "Formatted at 2026-09-25 07:34:08 on dist-test-slave-57fn"
server_key: "34ab6e288c1f35539d4ea06afc65a48c"
server_key_iv: "4fb79e72ca00484ad61393ade3aa0429"
server_key_version: "encryptionkey@0"
I20260925 07:34:08.827594 26845 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 07:34:08.828212 26860 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:34:08.828423 26845 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 07:34:08.828500 26845 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790321392278483-9480-0/minicluster-data/ts-2/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790321392278483-9480-0/minicluster-data/ts-2/wal
uuid: "51fb8e311c694370a5aa02d0e6411927"
format_stamp: "Formatted at 2026-09-25 07:34:08 on dist-test-slave-57fn"
server_key: "34ab6e288c1f35539d4ea06afc65a48c"
server_key_iv: "4fb79e72ca00484ad61393ade3aa0429"
server_key_version: "encryptionkey@0"
I20260925 07:34:08.828548 26845 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790321392278483-9480-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790321392278483-9480-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790321392278483-9480-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:34:08.853169 26845 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:34:08.855350 26845 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:34:08.855450 26845 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:34:08.855612 26845 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:34:08.855892 26845 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:34:08.855923 26845 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:34:08.855943 26845 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:34:08.856002 26845 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:34:08.861413 26845 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.3:41119
I20260925 07:34:08.861482 26973 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.3:41119 every 8 connection(s)
I20260925 07:34:08.861766 26845 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790321392278483-9480-0/minicluster-data/ts-2/data/info.pb
I20260925 07:34:08.866417 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 26845
I20260925 07:34:08.866505 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790321392278483-9480-0/minicluster-data/ts-2/wal/instance
I20260925 07:34:08.866734 9480 external_mini_cluster.cc:1428] Setting key 1e814402a6351f79b7648a40d64f8ea6
Sep 25 07:34:08 dist-test-slave-57fn krb5kdc[26470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321648, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.3@KRBTEST.COM for kudu/127.9.66.62@KRBTEST.COM
I20260925 07:34:08.870541 26572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:08.862514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.3:60277 (local address 127.9.66.62:42049)
0925 07:34:08.862564 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:08.862566 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:08.862760 (+ 194us) server_negotiation.cc:407] Connection header received
0925 07:34:08.863184 (+ 424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:08.863185 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:08.863196 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:08.863245 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:34:08.864078 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:08.864559 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:08.865401 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:08.865570 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:08.865613 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:08.867746 (+ 2133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:34:08.867747 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:34:08.867748 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:34:08.867755 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:34:08.869120 (+ 1365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:34:08.869564 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:34:08.869565 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:34:08.869565 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:34:08.869623 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:34:08.869920 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:34:08.869920 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:34:08.869921 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:34:08.870031 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:34:08.870105 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:08.870367 (+ 262us) server_negotiation.cc:299] Negotiation successful
0925 07:34:08.870445 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 07:34:08.870684 26976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:08.862498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:42049 (local address 127.9.66.3:60277)
0925 07:34:08.862652 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 07:34:08.862682 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0925 07:34:08.863091 (+ 409us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:34:08.863307 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:34:08.863310 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:34:08.863835 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:34:08.864005 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:08.864009 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:08.864640 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:08.864640 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:08.865327 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:08.865329 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:08.865633 (+ 304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:08.865634 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:08.865723 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:08.866093 (+ 370us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:34:08.866100 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:34:08.867649 (+ 1549us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:34:08.869218 (+ 1569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:34:08.869224 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:34:08.869226 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:34:08.869436 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:34:08.869687 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:34:08.869688 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:34:08.869688 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:34:08.869824 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:34:08.870096 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:34:08.870102 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:34:08.870232 (+ 130us) client_negotiation.cc:769] Sending connection context
0925 07:34:08.870330 (+ 98us) client_negotiation.cc:240] Negotiation successful
0925 07:34:08.870473 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":79,"threads_started":1}
I20260925 07:34:08.871135 26974 heartbeater.cc:347] Connected to a master server at 127.9.66.62:42049
I20260925 07:34:08.871207 26974 heartbeater.cc:464] Registering TS with master...
I20260925 07:34:08.871390 26974 heartbeater.cc:511] Master 127.9.66.62:42049 requested a full tablet report, sending...
I20260925 07:34:08.871747 26515 ts_manager.cc:194] Registered new tserver with Master: 51fb8e311c694370a5aa02d0e6411927 (127.9.66.3:41119)
I20260925 07:34:08.872201 26515 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.66.3@KRBTEST.COM'} at 127.9.66.3:60277
I20260925 07:34:08.878501 9480 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 07:34:08.885200 26572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:08.879887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50992 (local address 127.9.66.62:42049)
0925 07:34:08.879946 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:08.879948 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:08.880073 (+ 125us) server_negotiation.cc:407] Connection header received
0925 07:34:08.880117 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:08.880117 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:08.880126 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:08.880199 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:34:08.880713 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:08.881163 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:08.881894 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:08.882063 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:08.882132 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:08.883177 (+ 1045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:34:08.883178 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:34:08.883178 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:34:08.883183 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:34:08.884304 (+ 1121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:34:08.884589 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:34:08.884590 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:34:08.884590 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:34:08.884641 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:34:08.884780 (+ 139us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:34:08.884780 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:34:08.884780 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:34:08.884904 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:34:08.884973 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:08.885057 (+ 84us) server_negotiation.cc:299] Negotiation successful
0925 07:34:08.885147 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 07:34:08.886586 26515 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50992:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 07:34:08.888195 26515 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 07:34:08.894546 26982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:08.890376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.2:41287 (local address 127.0.0.1:37652)
0925 07:34:08.890463 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0925 07:34:08.890497 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 07:34:08.890559 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:34:08.891132 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:34:08.891135 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:34:08.891137 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:34:08.891197 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:08.891199 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:08.892319 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:08.892320 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:08.893565 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:08.893568 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:08.894281 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:08.894282 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:08.894352 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:08.894363 (+ 11us) client_negotiation.cc:769] Sending connection context
0925 07:34:08.894427 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 07:34:08.894475 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67,"thread_start_us":51,"threads_started":1}
I20260925 07:34:08.894569 26985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:08.890532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37652 (local address 127.9.66.2:41287)
0925 07:34:08.890944 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:08.890948 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:08.890956 (+ 8us) server_negotiation.cc:407] Connection header received
0925 07:34:08.891021 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:08.891023 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:08.891037 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:08.891112 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:34:08.891295 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:08.892222 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:08.893640 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:08.894204 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:08.894282 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:08.894334 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:08.894440 (+ 106us) server_negotiation.cc:299] Negotiation successful
0925 07:34:08.894480 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":96,"threads_started":1}
I20260925 07:34:08.894671 26983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:08.890465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.1:37825 (local address 127.0.0.1:55970)
0925 07:34:08.890720 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0925 07:34:08.890730 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 07:34:08.890783 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:34:08.891140 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:34:08.891141 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:34:08.891143 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:34:08.891215 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:08.891217 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:08.892543 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:08.892544 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:08.893591 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:08.893592 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:08.894423 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:08.894424 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:08.894490 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:08.894493 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 07:34:08.894541 (+ 48us) client_negotiation.cc:240] Negotiation successful
0925 07:34:08.894584 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":50,"threads_started":1}
I20260925 07:34:08.894897 26986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:08.890666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55970 (local address 127.9.66.1:37825)
0925 07:34:08.890879 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:08.890884 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:08.890895 (+ 11us) server_negotiation.cc:407] Connection header received
0925 07:34:08.890933 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:08.890935 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:08.890950 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:08.891069 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:34:08.891358 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:08.892443 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:08.893672 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:08.894272 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:08.894334 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:08.894389 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:08.894772 (+ 383us) server_negotiation.cc:299] Negotiation successful
0925 07:34:08.894809 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":81,"threads_started":1}
I20260925 07:34:08.895354 26984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:08.890557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.3:41119 (local address 127.0.0.1:34204)
0925 07:34:08.891207 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0925 07:34:08.891217 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 07:34:08.891275 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:34:08.891505 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:34:08.891506 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:34:08.891508 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:34:08.891542 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:08.891543 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:08.892750 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:08.892751 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:08.893807 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:08.893810 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:08.894941 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:08.894942 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:08.895178 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:08.895182 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 07:34:08.895253 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 07:34:08.895290 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":638,"thread_start_us":49,"threads_started":1}
I20260925 07:34:08.895378 26987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:08.890695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34204 (local address 127.9.66.3:41119)
0925 07:34:08.890895 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:08.890898 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:08.891329 (+ 431us) server_negotiation.cc:407] Connection header received
0925 07:34:08.891364 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:08.891366 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:08.891380 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:08.891446 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:34:08.891674 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:08.892664 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:08.893896 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:08.894528 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:08.894580 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:08.895086 (+ 506us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:08.895265 (+ 179us) server_negotiation.cc:299] Negotiation successful
0925 07:34:08.895300 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":66,"threads_started":1}
I20260925 07:34:08.895277 26774 tablet_service.cc:1467] Processing CreateTablet for tablet 36c80e2169164d4eab178b08dae2c270 (DEFAULT_TABLE table=test-table [id=7592f73965f74901bb18e268c94bb06c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:34:08.895602 26774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36c80e2169164d4eab178b08dae2c270. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:34:08.895540 26640 tablet_service.cc:1467] Processing CreateTablet for tablet 36c80e2169164d4eab178b08dae2c270 (DEFAULT_TABLE table=test-table [id=7592f73965f74901bb18e268c94bb06c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:34:08.895814 26640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36c80e2169164d4eab178b08dae2c270. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:34:08.898114 26908 tablet_service.cc:1467] Processing CreateTablet for tablet 36c80e2169164d4eab178b08dae2c270 (DEFAULT_TABLE table=test-table [id=7592f73965f74901bb18e268c94bb06c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:34:08.898417 26908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36c80e2169164d4eab178b08dae2c270. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:34:08.898917 26988 tablet_bootstrap.cc:492] T 36c80e2169164d4eab178b08dae2c270 P d89b70fd747b44a5bef016c8379dd0e4: Bootstrap starting.
I20260925 07:34:08.899130 26989 tablet_bootstrap.cc:492] T 36c80e2169164d4eab178b08dae2c270 P b87af5315b1648c28d1ee73d9a0ba874: Bootstrap starting.
I20260925 07:34:08.899781 26989 tablet_bootstrap.cc:654] T 36c80e2169164d4eab178b08dae2c270 P b87af5315b1648c28d1ee73d9a0ba874: Neither blocks nor log segments found. Creating new log.
I20260925 07:34:08.900362 26989 log.cc:824] T 36c80e2169164d4eab178b08dae2c270 P b87af5315b1648c28d1ee73d9a0ba874: Log is configured to *not* fsync() on all Append() calls
I20260925 07:34:08.901185 26988 tablet_bootstrap.cc:654] T 36c80e2169164d4eab178b08dae2c270 P d89b70fd747b44a5bef016c8379dd0e4: Neither blocks nor log segments found. Creating new log.
I20260925 07:34:08.901753 26991 tablet_bootstrap.cc:492] T 36c80e2169164d4eab178b08dae2c270 P 51fb8e311c694370a5aa02d0e6411927: Bootstrap starting.
I20260925 07:34:08.901885 26988 log.cc:824] T 36c80e2169164d4eab178b08dae2c270 P d89b70fd747b44a5bef016c8379dd0e4: Log is configured to *not* fsync() on all Append() calls
I20260925 07:34:08.901919 26989 tablet_bootstrap.cc:492] T 36c80e2169164d4eab178b08dae2c270 P b87af5315b1648c28d1ee73d9a0ba874: No bootstrap required, opened a new log
I20260925 07:34:08.901999 26989 ts_tablet_manager.cc:1397] T 36c80e2169164d4eab178b08dae2c270 P b87af5315b1648c28d1ee73d9a0ba874: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 07:34:08.902364 26991 tablet_bootstrap.cc:654] T 36c80e2169164d4eab178b08dae2c270 P 51fb8e311c694370a5aa02d0e6411927: Neither blocks nor log segments found. Creating new log.
I20260925 07:34:08.902576 26988 tablet_bootstrap.cc:492] T 36c80e2169164d4eab178b08dae2c270 P d89b70fd747b44a5bef016c8379dd0e4: No bootstrap required, opened a new log
I20260925 07:34:08.902650 26988 ts_tablet_manager.cc:1397] T 36c80e2169164d4eab178b08dae2c270 P d89b70fd747b44a5bef016c8379dd0e4: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260925 07:34:08.902894 26991 log.cc:824] T 36c80e2169164d4eab178b08dae2c270 P 51fb8e311c694370a5aa02d0e6411927: Log is configured to *not* fsync() on all Append() calls
I20260925 07:34:08.903616 26991 tablet_bootstrap.cc:492] T 36c80e2169164d4eab178b08dae2c270 P 51fb8e311c694370a5aa02d0e6411927: No bootstrap required, opened a new log
I20260925 07:34:08.903680 26991 ts_tablet_manager.cc:1397] T 36c80e2169164d4eab178b08dae2c270 P 51fb8e311c694370a5aa02d0e6411927: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 07:34:08.904114 26989 raft_consensus.cc:348] T 36c80e2169164d4eab178b08dae2c270 P b87af5315b1648c28d1ee73d9a0ba874 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d89b70fd747b44a5bef016c8379dd0e4" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 41287 } } peers { permanent_uuid: "b87af5315b1648c28d1ee73d9a0ba874" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 37825 } } peers { permanent_uuid: "51fb8e311c694370a5aa02d0e6411927" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 41119 } }
I20260925 07:34:08.904284 26989 raft_consensus.cc:374] T 36c80e2169164d4eab178b08dae2c270 P b87af5315b1648c28d1ee73d9a0ba874 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:34:08.904325 26989 raft_consensus.cc:729] T 36c80e2169164d4eab178b08dae2c270 P b87af5315b1648c28d1ee73d9a0ba874 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b87af5315b1648c28d1ee73d9a0ba874, State: Initialized, Role: FOLLOWER
I20260925 07:34:08.904372 26988 raft_consensus.cc:348] T 36c80e2169164d4eab178b08dae2c270 P d89b70fd747b44a5bef016c8379dd0e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d89b70fd747b44a5bef016c8379dd0e4" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 41287 } } peers { permanent_uuid: "b87af5315b1648c28d1ee73d9a0ba874" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 37825 } } peers { permanent_uuid: "51fb8e311c694370a5aa02d0e6411927" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 41119 } }
I20260925 07:34:08.904466 26989 consensus_queue.cc:260] T 36c80e2169164d4eab178b08dae2c270 P b87af5315b1648c28d1ee73d9a0ba874 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d89b70fd747b44a5bef016c8379dd0e4" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 41287 } } peers { permanent_uuid: "b87af5315b1648c28d1ee73d9a0ba874" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 37825 } } peers { permanent_uuid: "51fb8e311c694370a5aa02d0e6411927" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 41119 } }
I20260925 07:34:08.904557 26988 raft_consensus.cc:374] T 36c80e2169164d4eab178b08dae2c270 P d89b70fd747b44a5bef016c8379dd0e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:34:08.904598 26988 raft_consensus.cc:729] T 36c80e2169164d4eab178b08dae2c270 P d89b70fd747b44a5bef016c8379dd0e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d89b70fd747b44a5bef016c8379dd0e4, State: Initialized, Role: FOLLOWER
I20260925 07:34:08.904691 26988 consensus_queue.cc:260] T 36c80e2169164d4eab178b08dae2c270 P d89b70fd747b44a5bef016c8379dd0e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d89b70fd747b44a5bef016c8379dd0e4" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 41287 } } peers { permanent_uuid: "b87af5315b1648c28d1ee73d9a0ba874" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 37825 } } peers { permanent_uuid: "51fb8e311c694370a5aa02d0e6411927" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 41119 } }
I20260925 07:34:08.904747 26989 ts_tablet_manager.cc:1428] T 36c80e2169164d4eab178b08dae2c270 P b87af5315b1648c28d1ee73d9a0ba874: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 07:34:08.904886 26706 heartbeater.cc:503] Master 127.9.66.62:42049 was elected leader, sending a full tablet report...
I20260925 07:34:08.905287 26988 ts_tablet_manager.cc:1428] T 36c80e2169164d4eab178b08dae2c270 P d89b70fd747b44a5bef016c8379dd0e4: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 07:34:08.905396 26840 heartbeater.cc:503] Master 127.9.66.62:42049 was elected leader, sending a full tablet report...
I20260925 07:34:08.905962 26991 raft_consensus.cc:348] T 36c80e2169164d4eab178b08dae2c270 P 51fb8e311c694370a5aa02d0e6411927 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d89b70fd747b44a5bef016c8379dd0e4" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 41287 } } peers { permanent_uuid: "b87af5315b1648c28d1ee73d9a0ba874" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 37825 } } peers { permanent_uuid: "51fb8e311c694370a5aa02d0e6411927" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 41119 } }
I20260925 07:34:08.906143 26991 raft_consensus.cc:374] T 36c80e2169164d4eab178b08dae2c270 P 51fb8e311c694370a5aa02d0e6411927 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:34:08.906190 26991 raft_consensus.cc:729] T 36c80e2169164d4eab178b08dae2c270 P 51fb8e311c694370a5aa02d0e6411927 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51fb8e311c694370a5aa02d0e6411927, State: Initialized, Role: FOLLOWER
I20260925 07:34:08.906307 26991 consensus_queue.cc:260] T 36c80e2169164d4eab178b08dae2c270 P 51fb8e311c694370a5aa02d0e6411927 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d89b70fd747b44a5bef016c8379dd0e4" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 41287 } } peers { permanent_uuid: "b87af5315b1648c28d1ee73d9a0ba874" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 37825 } } peers { permanent_uuid: "51fb8e311c694370a5aa02d0e6411927" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 41119 } }
I20260925 07:34:08.906587 26991 ts_tablet_manager.cc:1428] T 36c80e2169164d4eab178b08dae2c270 P 51fb8e311c694370a5aa02d0e6411927: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 07:34:08.906596 26974 heartbeater.cc:503] Master 127.9.66.62:42049 was elected leader, sending a full tablet report...
I20260925 07:34:08.933733 26996 raft_consensus.cc:482] T 36c80e2169164d4eab178b08dae2c270 P 51fb8e311c694370a5aa02d0e6411927 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 07:34:08.933811 26996 raft_consensus.cc:504] T 36c80e2169164d4eab178b08dae2c270 P 51fb8e311c694370a5aa02d0e6411927 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d89b70fd747b44a5bef016c8379dd0e4" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 41287 } } peers { permanent_uuid: "b87af5315b1648c28d1ee73d9a0ba874" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 37825 } } peers { permanent_uuid: "51fb8e311c694370a5aa02d0e6411927" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 41119 } }
I20260925 07:34:08.934072 26996 leader_election.cc:288] T 36c80e2169164d4eab178b08dae2c270 P 51fb8e311c694370a5aa02d0e6411927 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d89b70fd747b44a5bef016c8379dd0e4 (127.9.66.2:41287), b87af5315b1648c28d1ee73d9a0ba874 (127.9.66.1:37825)
I20260925 07:34:08.936723 26976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:08.934190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.2:41287 (local address 127.9.66.3:53991)
0925 07:34:08.934231 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0925 07:34:08.934241 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 07:34:08.934279 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:34:08.934418 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:34:08.934419 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:34:08.934420 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:34:08.934481 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:08.934482 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:08.935097 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:08.935099 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:08.935839 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:08.935841 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:08.936446 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:08.936447 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:08.936548 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:08.936552 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 07:34:08.936620 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 07:34:08.936658 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20}
I20260925 07:34:08.936721 26985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:08.934277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.3:53991 (local address 127.9.66.2:41287)
0925 07:34:08.934310 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:08.934312 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:08.934320 (+ 8us) server_negotiation.cc:407] Connection header received
0925 07:34:08.934340 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:08.934341 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:08.934349 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:08.934415 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:34:08.934546 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:08.934996 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:08.935930 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:08.936361 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:08.936434 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:08.936444 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:08.936625 (+ 181us) server_negotiation.cc:299] Negotiation successful
0925 07:34:08.936658 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 07:34:08.936842 26986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:08.934271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.3:34075 (local address 127.9.66.1:37825)
0925 07:34:08.934311 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:08.934313 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:08.934414 (+ 101us) server_negotiation.cc:407] Connection header received
0925 07:34:08.934468 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:08.934469 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:08.934479 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:08.934528 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:34:08.934670 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:08.935137 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:08.936042 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:08.936521 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:08.936597 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:08.936605 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:08.936748 (+ 143us) server_negotiation.cc:299] Negotiation successful
0925 07:34:08.936782 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 07:34:08.936859 26997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:08.934211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.1:37825 (local address 127.9.66.3:34075)
0925 07:34:08.934344 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0925 07:34:08.934352 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 07:34:08.934412 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:34:08.934531 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:34:08.934532 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:34:08.934533 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:34:08.934597 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:08.934598 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:08.935218 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:08.935219 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:08.935952 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:08.935954 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:08.936596 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:08.936597 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:08.936668 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:08.936672 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 07:34:08.936744 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 07:34:08.936781 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":81,"threads_started":1}
I20260925 07:34:08.937057 26794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36c80e2169164d4eab178b08dae2c270" candidate_uuid: "51fb8e311c694370a5aa02d0e6411927" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d89b70fd747b44a5bef016c8379dd0e4" is_pre_election: true
I20260925 07:34:08.937175 26794 raft_consensus.cc:2454] T 36c80e2169164d4eab178b08dae2c270 P d89b70fd747b44a5bef016c8379dd0e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51fb8e311c694370a5aa02d0e6411927 in term 0.
I20260925 07:34:08.937125 26660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36c80e2169164d4eab178b08dae2c270" candidate_uuid: "51fb8e311c694370a5aa02d0e6411927" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b87af5315b1648c28d1ee73d9a0ba874" is_pre_election: true
I20260925 07:34:08.937283 26660 raft_consensus.cc:2454] T 36c80e2169164d4eab178b08dae2c270 P b87af5315b1648c28d1ee73d9a0ba874 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51fb8e311c694370a5aa02d0e6411927 in term 0.
I20260925 07:34:08.937390 26862 leader_election.cc:302] T 36c80e2169164d4eab178b08dae2c270 P 51fb8e311c694370a5aa02d0e6411927 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51fb8e311c694370a5aa02d0e6411927, d89b70fd747b44a5bef016c8379dd0e4; no voters:
I20260925 07:34:08.937546 26996 raft_consensus.cc:2781] T 36c80e2169164d4eab178b08dae2c270 P 51fb8e311c694370a5aa02d0e6411927 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 07:34:08.937601 26996 raft_consensus.cc:482] T 36c80e2169164d4eab178b08dae2c270 P 51fb8e311c694370a5aa02d0e6411927 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 07:34:08.937637 26996 raft_consensus.cc:3037] T 36c80e2169164d4eab178b08dae2c270 P 51fb8e311c694370a5aa02d0e6411927 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:34:08.938261 26996 raft_consensus.cc:504] T 36c80e2169164d4eab178b08dae2c270 P 51fb8e311c694370a5aa02d0e6411927 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d89b70fd747b44a5bef016c8379dd0e4" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 41287 } } peers { permanent_uuid: "b87af5315b1648c28d1ee73d9a0ba874" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 37825 } } peers { permanent_uuid: "51fb8e311c694370a5aa02d0e6411927" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 41119 } }
I20260925 07:34:08.938453 26996 leader_election.cc:288] T 36c80e2169164d4eab178b08dae2c270 P 51fb8e311c694370a5aa02d0e6411927 [CANDIDATE]: Term 1 election: Requested vote from peers d89b70fd747b44a5bef016c8379dd0e4 (127.9.66.2:41287), b87af5315b1648c28d1ee73d9a0ba874 (127.9.66.1:37825)
I20260925 07:34:08.938511 26794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36c80e2169164d4eab178b08dae2c270" candidate_uuid: "51fb8e311c694370a5aa02d0e6411927" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d89b70fd747b44a5bef016c8379dd0e4"
I20260925 07:34:08.938551 26660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36c80e2169164d4eab178b08dae2c270" candidate_uuid: "51fb8e311c694370a5aa02d0e6411927" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b87af5315b1648c28d1ee73d9a0ba874"
I20260925 07:34:08.938583 26794 raft_consensus.cc:3037] T 36c80e2169164d4eab178b08dae2c270 P d89b70fd747b44a5bef016c8379dd0e4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:34:08.938632 26660 raft_consensus.cc:3037] T 36c80e2169164d4eab178b08dae2c270 P b87af5315b1648c28d1ee73d9a0ba874 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:34:08.939224 26794 raft_consensus.cc:2454] T 36c80e2169164d4eab178b08dae2c270 P d89b70fd747b44a5bef016c8379dd0e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51fb8e311c694370a5aa02d0e6411927 in term 1.
I20260925 07:34:08.939297 26660 raft_consensus.cc:2454] T 36c80e2169164d4eab178b08dae2c270 P b87af5315b1648c28d1ee73d9a0ba874 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51fb8e311c694370a5aa02d0e6411927 in term 1.
I20260925 07:34:08.939365 26862 leader_election.cc:302] T 36c80e2169164d4eab178b08dae2c270 P 51fb8e311c694370a5aa02d0e6411927 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51fb8e311c694370a5aa02d0e6411927, d89b70fd747b44a5bef016c8379dd0e4; no voters:
I20260925 07:34:08.939455 26996 raft_consensus.cc:2781] T 36c80e2169164d4eab178b08dae2c270 P 51fb8e311c694370a5aa02d0e6411927 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:34:08.939620 26996 raft_consensus.cc:686] T 36c80e2169164d4eab178b08dae2c270 P 51fb8e311c694370a5aa02d0e6411927 [term 1 LEADER]: Becoming Leader. State: Replica: 51fb8e311c694370a5aa02d0e6411927, State: Running, Role: LEADER
I20260925 07:34:08.939714 26996 consensus_queue.cc:237] T 36c80e2169164d4eab178b08dae2c270 P 51fb8e311c694370a5aa02d0e6411927 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d89b70fd747b44a5bef016c8379dd0e4" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 41287 } } peers { permanent_uuid: "b87af5315b1648c28d1ee73d9a0ba874" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 37825 } } peers { permanent_uuid: "51fb8e311c694370a5aa02d0e6411927" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 41119 } }
I20260925 07:34:08.940567 26514 catalog_manager.cc:5495] T 36c80e2169164d4eab178b08dae2c270 P 51fb8e311c694370a5aa02d0e6411927 reported cstate change: term changed from 0 to 1, leader changed from <none> to 51fb8e311c694370a5aa02d0e6411927 (127.9.66.3). New cstate: current_term: 1 leader_uuid: "51fb8e311c694370a5aa02d0e6411927" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d89b70fd747b44a5bef016c8379dd0e4" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 41287 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b87af5315b1648c28d1ee73d9a0ba874" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 37825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51fb8e311c694370a5aa02d0e6411927" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 41119 } health_report { overall_health: HEALTHY } } }
I20260925 07:34:08.953311 26987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:08.951228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34210 (local address 127.9.66.3:41119)
0925 07:34:08.951404 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:08.951405 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:08.951413 (+ 8us) server_negotiation.cc:407] Connection header received
0925 07:34:08.951444 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:08.951445 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:08.951458 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:08.951513 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 07:34:08.951622 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:08.952087 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:08.952752 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:08.952913 (+ 161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:08.952985 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:08.953090 (+ 105us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 07:34:08.953164 (+ 74us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 07:34:08.953217 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:08.953243 (+ 26us) server_negotiation.cc:299] Negotiation successful
0925 07:34:08.953257 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158}
I20260925 07:34:08.954746 26794 raft_consensus.cc:1261] T 36c80e2169164d4eab178b08dae2c270 P d89b70fd747b44a5bef016c8379dd0e4 [term 1 FOLLOWER]: Refusing update from remote peer 51fb8e311c694370a5aa02d0e6411927: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:34:08.954770 26660 raft_consensus.cc:1261] T 36c80e2169164d4eab178b08dae2c270 P b87af5315b1648c28d1ee73d9a0ba874 [term 1 FOLLOWER]: Refusing update from remote peer 51fb8e311c694370a5aa02d0e6411927: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:34:08.955015 26998 consensus_queue.cc:1035] T 36c80e2169164d4eab178b08dae2c270 P 51fb8e311c694370a5aa02d0e6411927 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d89b70fd747b44a5bef016c8379dd0e4" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 41287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:34:08.955122 26996 consensus_queue.cc:1035] T 36c80e2169164d4eab178b08dae2c270 P 51fb8e311c694370a5aa02d0e6411927 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b87af5315b1648c28d1ee73d9a0ba874" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 37825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 07:34:08.956671 26841 tablet.cc:2369] T 36c80e2169164d4eab178b08dae2c270 P d89b70fd747b44a5bef016c8379dd0e4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 07:34:08.957625 27003 mvcc.cc:205] Tried to move back new op lower bound from 7333157474116653056 to 7333157474057981952. Current Snapshot: MvccSnapshot[applied={T|T < 7333157474116653056}]
I20260925 07:34:08.959261 26514 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50992:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 07:34:08.959367 26514 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50992:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 07:34:08.960062 26514 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d0adaabfb25a41eaadf4baf35e082c22: Sending DeleteTablet for 3 replicas of tablet 36c80e2169164d4eab178b08dae2c270
I20260925 07:34:08.960405 26908 tablet_service.cc:1514] Processing DeleteTablet for tablet 36c80e2169164d4eab178b08dae2c270 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:34:08 UTC) from {username='kudu', principal='kudu/127.9.66.62@KRBTEST.COM'} at 127.0.0.1:34204
I20260925 07:34:08.960450 26774 tablet_service.cc:1514] Processing DeleteTablet for tablet 36c80e2169164d4eab178b08dae2c270 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:34:08 UTC) from {username='kudu', principal='kudu/127.9.66.62@KRBTEST.COM'} at 127.0.0.1:37652
I20260925 07:34:08.960453 26640 tablet_service.cc:1514] Processing DeleteTablet for tablet 36c80e2169164d4eab178b08dae2c270 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:34:08 UTC) from {username='kudu', principal='kudu/127.9.66.62@KRBTEST.COM'} at 127.0.0.1:55970
I20260925 07:34:08.960654 27010 tablet_replica.cc:330] T 36c80e2169164d4eab178b08dae2c270 P 51fb8e311c694370a5aa02d0e6411927: stopping tablet replica
I20260925 07:34:08.960750 27010 raft_consensus.cc:2229] T 36c80e2169164d4eab178b08dae2c270 P 51fb8e311c694370a5aa02d0e6411927 [term 1 LEADER]: Raft consensus shutting down.
I20260925 07:34:08.960764 27012 tablet_replica.cc:330] T 36c80e2169164d4eab178b08dae2c270 P b87af5315b1648c28d1ee73d9a0ba874: stopping tablet replica
I20260925 07:34:08.960812 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 26577
I20260925 07:34:08.960870 27010 raft_consensus.cc:2258] T 36c80e2169164d4eab178b08dae2c270 P 51fb8e311c694370a5aa02d0e6411927 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:34:08.961560 27011 tablet_replica.cc:330] T 36c80e2169164d4eab178b08dae2c270 P d89b70fd747b44a5bef016c8379dd0e4: stopping tablet replica
I20260925 07:34:08.961640 27011 raft_consensus.cc:2229] T 36c80e2169164d4eab178b08dae2c270 P d89b70fd747b44a5bef016c8379dd0e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 07:34:08.961717 27011 raft_consensus.cc:2258] T 36c80e2169164d4eab178b08dae2c270 P d89b70fd747b44a5bef016c8379dd0e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:34:08.962154 27010 ts_tablet_manager.cc:1905] T 36c80e2169164d4eab178b08dae2c270 P 51fb8e311c694370a5aa02d0e6411927: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 07:34:08.962610 27011 ts_tablet_manager.cc:1905] T 36c80e2169164d4eab178b08dae2c270 P d89b70fd747b44a5bef016c8379dd0e4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 07:34:08.963999 27010 ts_tablet_manager.cc:1918] T 36c80e2169164d4eab178b08dae2c270 P 51fb8e311c694370a5aa02d0e6411927: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 07:34:08.964057 27011 ts_tablet_manager.cc:1918] T 36c80e2169164d4eab178b08dae2c270 P d89b70fd747b44a5bef016c8379dd0e4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 07:34:08.964077 27010 log.cc:1196] T 36c80e2169164d4eab178b08dae2c270 P 51fb8e311c694370a5aa02d0e6411927: Deleting WAL directory at /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790321392278483-9480-0/minicluster-data/ts-2/wal/wals/36c80e2169164d4eab178b08dae2c270
I20260925 07:34:08.964121 27011 log.cc:1196] T 36c80e2169164d4eab178b08dae2c270 P d89b70fd747b44a5bef016c8379dd0e4: Deleting WAL directory at /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790321392278483-9480-0/minicluster-data/ts-1/wal/wals/36c80e2169164d4eab178b08dae2c270
I20260925 07:34:08.964376 27010 ts_tablet_manager.cc:1939] T 36c80e2169164d4eab178b08dae2c270 P 51fb8e311c694370a5aa02d0e6411927: Deleting consensus metadata
I20260925 07:34:08.964435 27011 ts_tablet_manager.cc:1939] T 36c80e2169164d4eab178b08dae2c270 P d89b70fd747b44a5bef016c8379dd0e4: Deleting consensus metadata
I20260925 07:34:08.965173 26500 catalog_manager.cc:4841] TS d89b70fd747b44a5bef016c8379dd0e4 (127.9.66.2:41287): tablet 36c80e2169164d4eab178b08dae2c270 (table test-table [id=7592f73965f74901bb18e268c94bb06c]) successfully deleted
I20260925 07:34:08.965319 26500 catalog_manager.cc:4841] TS 51fb8e311c694370a5aa02d0e6411927 (127.9.66.3:41119): tablet 36c80e2169164d4eab178b08dae2c270 (table test-table [id=7592f73965f74901bb18e268c94bb06c]) successfully deleted
W20260925 07:34:08.967582 26502 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.66.1:37825 (error 108)
I20260925 07:34:08.968077 26984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:08.967942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.1:37825 (local address 127.0.0.1:55986)
0925 07:34:08.968001 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0925 07:34:08.968043 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.1:37825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260925 07:34:08.968142 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 26711
W20260925 07:34:08.975167 26502 catalog_manager.cc:4584] TS b87af5315b1648c28d1ee73d9a0ba874 (127.9.66.1:37825): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 36c80e2169164d4eab178b08dae2c270: Network error: Client connection negotiation failed: client connection to 127.9.66.1:37825: connect: Connection refused (error 111)
I20260925 07:34:08.975279 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 26845
I20260925 07:34:08.979978 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 26483
2026-09-25T07:34:08Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (1869 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-25T07:34:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T07:34:08Z Disabled control of system clock
I20260925 07:34:09.006671 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790321392278483-9480-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790321392278483-9480-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790321392278483-9480-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.66.62:34249
--webserver_interface=127.9.66.62
--webserver_port=0
--builtin_ntp_servers=127.9.66.20:43747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.66.62:34249
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 07:34:09.083700 27017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:34:09.083887 27017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:34:09.083909 27017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:34:09.084936 27017 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:34:09.084968 27017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:34:09.084981 27017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:34:09.084993 27017 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:34:09.085024 27017 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:34:09.086273 27017 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:43747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790321392278483-9480-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790321392278483-9480-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.66.62:34249
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.66.62:34249
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.66.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27017
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790321392278483-9480-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:34:09.086439 27017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:34:09.086647 27017 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:34:09.091303 27022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:34:09.091277 27025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:34:09.091463 27017 server_base.cc:1034] running on GCE node
W20260925 07:34:09.091281 27023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:34:09.091760 27017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:34:09.092051 27017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:34:09.093219 27017 hybrid_clock.cc:648] HybridClock initialized: now 1790321649093200 us; error 30 us; skew 500 ppm
I20260925 07:34:09.094554 27017 webserver.cc:460] Webserver started at http://127.9.66.62:35021/ using document root <none> and password file <none>
I20260925 07:34:09.094815 27017 fs_manager.cc:359] Metadata directory not provided
I20260925 07:34:09.094897 27017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:34:09.095031 27017 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:34:09.096017 27017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790321392278483-9480-0/minicluster-data/master-0/data/instance:
uuid: "03d9b59ddca741b888127e072d46371d"
format_stamp: "Formatted at 2026-09-25 07:34:09 on dist-test-slave-57fn"
server_key: "0255b14cb609669c0425eb46045a5c7a"
server_key_iv: "32d514124e1ca92b279af9a570c19638"
server_key_version: "encryptionkey@0"
I20260925 07:34:09.096393 27017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790321392278483-9480-0/minicluster-data/master-0/wal/instance:
uuid: "03d9b59ddca741b888127e072d46371d"
format_stamp: "Formatted at 2026-09-25 07:34:09 on dist-test-slave-57fn"
server_key: "0255b14cb609669c0425eb46045a5c7a"
server_key_iv: "32d514124e1ca92b279af9a570c19638"
server_key_version: "encryptionkey@0"
I20260925 07:34:09.097652 27017 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 07:34:09.098357 27031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:34:09.098513 27017 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 07:34:09.098598 27017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790321392278483-9480-0/minicluster-data/master-0/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790321392278483-9480-0/minicluster-data/master-0/wal
uuid: "03d9b59ddca741b888127e072d46371d"
format_stamp: "Formatted at 2026-09-25 07:34:09 on dist-test-slave-57fn"
server_key: "0255b14cb609669c0425eb46045a5c7a"
server_key_iv: "32d514124e1ca92b279af9a570c19638"
server_key_version: "encryptionkey@0"
I20260925 07:34:09.098675 27017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790321392278483-9480-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790321392278483-9480-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790321392278483-9480-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:34:09.110502 27017 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:34:09.110756 27017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:34:09.110898 27017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:34:09.114533 27017 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.62:34249
I20260925 07:34:09.114563 27083 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.62:34249 every 8 connection(s)
I20260925 07:34:09.114883 27017 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
I20260925 07:34:09.115757 27084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:34:09.117933 27084 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03d9b59ddca741b888127e072d46371d: Bootstrap starting.
I20260925 07:34:09.118450 27084 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 03d9b59ddca741b888127e072d46371d: Neither blocks nor log segments found. Creating new log.
I20260925 07:34:09.118719 27084 log.cc:824] T 00000000000000000000000000000000 P 03d9b59ddca741b888127e072d46371d: Log is configured to *not* fsync() on all Append() calls
I20260925 07:34:09.119330 27084 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03d9b59ddca741b888127e072d46371d: No bootstrap required, opened a new log
I20260925 07:34:09.120759 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 27017
I20260925 07:34:09.120854 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790321392278483-9480-0/minicluster-data/master-0/wal/instance
I20260925 07:34:09.120924 27084 raft_consensus.cc:348] T 00000000000000000000000000000000 P 03d9b59ddca741b888127e072d46371d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03d9b59ddca741b888127e072d46371d" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 34249 } }
I20260925 07:34:09.121060 9480 external_mini_cluster.cc:1428] Setting key 287f9b669c234cb62e0fc16c2e707650
I20260925 07:34:09.121083 27084 raft_consensus.cc:374] T 00000000000000000000000000000000 P 03d9b59ddca741b888127e072d46371d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:34:09.121121 27084 raft_consensus.cc:729] T 00000000000000000000000000000000 P 03d9b59ddca741b888127e072d46371d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03d9b59ddca741b888127e072d46371d, State: Initialized, Role: FOLLOWER
I20260925 07:34:09.121251 27084 consensus_queue.cc:260] T 00000000000000000000000000000000 P 03d9b59ddca741b888127e072d46371d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03d9b59ddca741b888127e072d46371d" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 34249 } }
I20260925 07:34:09.121310 27084 raft_consensus.cc:388] T 00000000000000000000000000000000 P 03d9b59ddca741b888127e072d46371d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 07:34:09.121351 27084 raft_consensus.cc:482] T 00000000000000000000000000000000 P 03d9b59ddca741b888127e072d46371d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 07:34:09.121404 27084 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 03d9b59ddca741b888127e072d46371d [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:34:09.122011 27084 raft_consensus.cc:504] T 00000000000000000000000000000000 P 03d9b59ddca741b888127e072d46371d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03d9b59ddca741b888127e072d46371d" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 34249 } }
I20260925 07:34:09.122103 27084 leader_election.cc:302] T 00000000000000000000000000000000 P 03d9b59ddca741b888127e072d46371d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 03d9b59ddca741b888127e072d46371d; no voters:
I20260925 07:34:09.122268 27084 leader_election.cc:288] T 00000000000000000000000000000000 P 03d9b59ddca741b888127e072d46371d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 07:34:09.122295 27089 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 03d9b59ddca741b888127e072d46371d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:34:09.122431 27089 raft_consensus.cc:686] T 00000000000000000000000000000000 P 03d9b59ddca741b888127e072d46371d [term 1 LEADER]: Becoming Leader. State: Replica: 03d9b59ddca741b888127e072d46371d, State: Running, Role: LEADER
I20260925 07:34:09.122478 27084 sys_catalog.cc:564] T 00000000000000000000000000000000 P 03d9b59ddca741b888127e072d46371d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:34:09.122509 27089 consensus_queue.cc:237] T 00000000000000000000000000000000 P 03d9b59ddca741b888127e072d46371d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03d9b59ddca741b888127e072d46371d" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 34249 } }
I20260925 07:34:09.122825 27091 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03d9b59ddca741b888127e072d46371d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "03d9b59ddca741b888127e072d46371d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03d9b59ddca741b888127e072d46371d" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 34249 } } }
I20260925 07:34:09.122815 27089 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03d9b59ddca741b888127e072d46371d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 03d9b59ddca741b888127e072d46371d. Latest consensus state: current_term: 1 leader_uuid: "03d9b59ddca741b888127e072d46371d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03d9b59ddca741b888127e072d46371d" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 34249 } } }
I20260925 07:34:09.122992 27091 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03d9b59ddca741b888127e072d46371d [sys.catalog]: This master's current role is: LEADER
I20260925 07:34:09.123004 27089 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03d9b59ddca741b888127e072d46371d [sys.catalog]: This master's current role is: LEADER
I20260925 07:34:09.123791 27098 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 07:34:09.124451 27098 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 07:34:09.125805 27098 catalog_manager.cc:1347] Generated new cluster ID: a3c624ab930345ed9e5e03e0f8ae3529
I20260925 07:34:09.125854 27098 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 07:34:09.127378 27088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:09.121570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35470 (local address 127.9.66.62:34249)
0925 07:34:09.121809 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:09.121817 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:09.121852 (+ 35us) server_negotiation.cc:407] Connection header received
0925 07:34:09.121952 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:09.121957 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:09.122010 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:09.122154 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:34:09.122419 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:09.124173 (+ 1754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:09.125168 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:09.125717 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:09.125824 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:09.126688 (+ 864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:34:09.126692 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:34:09.126693 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 07:34:09.126888 (+ 195us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:34:09.126927 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 07:34:09.126940 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:34:09.127001 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:09.127121 (+ 120us) server_negotiation.cc:299] Negotiation successful
0925 07:34:09.127189 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":97,"threads_started":1}
I20260925 07:34:09.129876 27098 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 07:34:09.130316 27098 catalog_manager.cc:1504] Loading token signing keys...
I20260925 07:34:09.132848 27098 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 03d9b59ddca741b888127e072d46371d: Generated new TSK 0
I20260925 07:34:09.133030 27098 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 07:34:09.135777 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790321392278483-9480-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790321392278483-9480-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790321392278483-9480-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790321392278483-9480-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.66.1:0
--local_ip_for_outbound_sockets=127.9.66.1
--webserver_interface=127.9.66.1
--webserver_port=0
--tserver_master_addrs=127.9.66.62:34249
--builtin_ntp_servers=127.9.66.20:43747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 07:34:09.215613 27108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:34:09.215811 27108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:34:09.215831 27108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:34:09.216970 27108 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:34:09.217008 27108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:34:09.217033 27108 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.66.1
I20260925 07:34:09.218405 27108 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:43747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790321392278483-9480-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790321392278483-9480-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.66.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790321392278483-9480-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.66.1
--webserver_port=0
--tserver_master_addrs=127.9.66.62:34249
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27108
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.66.1
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790321392278483-9480-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:34:09.218618 27108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:34:09.218910 27108 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:34:09.224800 27116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:34:09.224800 27114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:34:09.224825 27113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:34:09.225277 27108 server_base.cc:1034] running on GCE node
I20260925 07:34:09.225474 27108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:34:09.225741 27108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:34:09.226922 27108 hybrid_clock.cc:648] HybridClock initialized: now 1790321649226910 us; error 29 us; skew 500 ppm
I20260925 07:34:09.228195 27108 webserver.cc:460] Webserver started at http://127.9.66.1:36411/ using document root <none> and password file <none>
I20260925 07:34:09.228487 27108 fs_manager.cc:359] Metadata directory not provided
I20260925 07:34:09.228555 27108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:34:09.228709 27108 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:34:09.230150 27108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790321392278483-9480-0/minicluster-data/ts-0/data/instance:
uuid: "2d71fd955d5044c1b3dae2fc6f54d59e"
format_stamp: "Formatted at 2026-09-25 07:34:09 on dist-test-slave-57fn"
server_key: "48a135a22551bd3940e07f0bacb8723c"
server_key_iv: "ed1f3ddb1deecf9363159a8ac97fdfa0"
server_key_version: "encryptionkey@0"
I20260925 07:34:09.230525 27108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790321392278483-9480-0/minicluster-data/ts-0/wal/instance:
uuid: "2d71fd955d5044c1b3dae2fc6f54d59e"
format_stamp: "Formatted at 2026-09-25 07:34:09 on dist-test-slave-57fn"
server_key: "48a135a22551bd3940e07f0bacb8723c"
server_key_iv: "ed1f3ddb1deecf9363159a8ac97fdfa0"
server_key_version: "encryptionkey@0"
I20260925 07:34:09.231873 27108 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 07:34:09.232743 27122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:34:09.232916 27108 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:34:09.233004 27108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790321392278483-9480-0/minicluster-data/ts-0/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790321392278483-9480-0/minicluster-data/ts-0/wal
uuid: "2d71fd955d5044c1b3dae2fc6f54d59e"
format_stamp: "Formatted at 2026-09-25 07:34:09 on dist-test-slave-57fn"
server_key: "48a135a22551bd3940e07f0bacb8723c"
server_key_iv: "ed1f3ddb1deecf9363159a8ac97fdfa0"
server_key_version: "encryptionkey@0"
I20260925 07:34:09.233084 27108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790321392278483-9480-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790321392278483-9480-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790321392278483-9480-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:34:09.252144 27108 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:34:09.252460 27108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:34:09.252612 27108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:34:09.252832 27108 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:34:09.253171 27108 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:34:09.253224 27108 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:34:09.253263 27108 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:34:09.253290 27108 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:34:09.258853 27108 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.1:33225
I20260925 07:34:09.258904 27235 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.1:33225 every 8 connection(s)
I20260925 07:34:09.259205 27108 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790321392278483-9480-0/minicluster-data/ts-0/data/info.pb
I20260925 07:34:09.260485 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 27108
I20260925 07:34:09.260604 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790321392278483-9480-0/minicluster-data/ts-0/wal/instance
I20260925 07:34:09.260799 9480 external_mini_cluster.cc:1428] Setting key 628b1f880f7b97136aca552186925816
I20260925 07:34:09.261807 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790321392278483-9480-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790321392278483-9480-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790321392278483-9480-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790321392278483-9480-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.66.2:0
--local_ip_for_outbound_sockets=127.9.66.2
--webserver_interface=127.9.66.2
--webserver_port=0
--tserver_master_addrs=127.9.66.62:34249
--builtin_ntp_servers=127.9.66.20:43747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 07:34:09.265674 27088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:09.259893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.1:55323 (local address 127.9.66.62:34249)
0925 07:34:09.259980 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:09.259982 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:09.260211 (+ 229us) server_negotiation.cc:407] Connection header received
0925 07:34:09.260944 (+ 733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:09.260945 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:09.260959 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:09.261052 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:34:09.261880 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:09.262682 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:09.264128 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:09.264320 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:09.264422 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:09.265379 (+ 957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:34:09.265380 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:34:09.265381 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 07:34:09.265388 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:34:09.265404 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 07:34:09.265408 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:34:09.265466 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:09.265585 (+ 119us) server_negotiation.cc:299] Negotiation successful
0925 07:34:09.265613 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 07:34:09.265967 27238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:09.259878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34249 (local address 127.9.66.1:55323)
0925 07:34:09.260082 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0925 07:34:09.260121 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 07:34:09.260811 (+ 690us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:34:09.261139 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:34:09.261144 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:34:09.261153 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:34:09.261789 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:09.261794 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:09.262774 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:09.262775 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:09.264030 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:09.264033 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:09.264448 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:09.264449 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:09.264606 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:09.265181 (+ 575us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 07:34:09.265181 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:34:09.265220 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 07:34:09.265250 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:34:09.265481 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:34:09.265487 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:34:09.265490 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 07:34:09.265586 (+ 96us) client_negotiation.cc:240] Negotiation successful
0925 07:34:09.265702 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":101,"threads_started":1}
I20260925 07:34:09.266598 27236 heartbeater.cc:347] Connected to a master server at 127.9.66.62:34249
I20260925 07:34:09.266700 27236 heartbeater.cc:464] Registering TS with master...
I20260925 07:34:09.266907 27236 heartbeater.cc:511] Master 127.9.66.62:34249 requested a full tablet report, sending...
I20260925 07:34:09.267449 27048 ts_manager.cc:194] Registered new tserver with Master: 2d71fd955d5044c1b3dae2fc6f54d59e (127.9.66.1:33225)
I20260925 07:34:09.268064 27048 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.66.1:55323
W20260925 07:34:09.342475 27239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:34:09.342687 27239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:34:09.342708 27239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:34:09.343842 27239 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:34:09.343879 27239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:34:09.343904 27239 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.66.2
I20260925 07:34:09.345237 27239 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:43747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790321392278483-9480-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790321392278483-9480-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.66.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790321392278483-9480-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.66.2
--webserver_port=0
--tserver_master_addrs=127.9.66.62:34249
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27239
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.66.2
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790321392278483-9480-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:34:09.345539 27239 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:34:09.345836 27239 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 07:34:09.351256 27239 server_base.cc:1034] running on GCE node
W20260925 07:34:09.351217 27244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:34:09.351239 27245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:34:09.351238 27247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:34:09.351663 27239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:34:09.351948 27239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:34:09.353099 27239 hybrid_clock.cc:648] HybridClock initialized: now 1790321649353085 us; error 30 us; skew 500 ppm
I20260925 07:34:09.354525 27239 webserver.cc:460] Webserver started at http://127.9.66.2:45163/ using document root <none> and password file <none>
I20260925 07:34:09.354774 27239 fs_manager.cc:359] Metadata directory not provided
I20260925 07:34:09.354821 27239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:34:09.355003 27239 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:34:09.355962 27239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790321392278483-9480-0/minicluster-data/ts-1/data/instance:
uuid: "9a8822c17cdb45b5b619c7a2fad59cd6"
format_stamp: "Formatted at 2026-09-25 07:34:09 on dist-test-slave-57fn"
server_key: "88c13d5587682ec527dfb8bb1dfdc64c"
server_key_iv: "3765700e08722f0c3a4262cd78775322"
server_key_version: "encryptionkey@0"
I20260925 07:34:09.356292 27239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790321392278483-9480-0/minicluster-data/ts-1/wal/instance:
uuid: "9a8822c17cdb45b5b619c7a2fad59cd6"
format_stamp: "Formatted at 2026-09-25 07:34:09 on dist-test-slave-57fn"
server_key: "88c13d5587682ec527dfb8bb1dfdc64c"
server_key_iv: "3765700e08722f0c3a4262cd78775322"
server_key_version: "encryptionkey@0"
I20260925 07:34:09.357726 27239 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 07:34:09.358369 27253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:34:09.358654 27239 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 07:34:09.358789 27239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790321392278483-9480-0/minicluster-data/ts-1/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790321392278483-9480-0/minicluster-data/ts-1/wal
uuid: "9a8822c17cdb45b5b619c7a2fad59cd6"
format_stamp: "Formatted at 2026-09-25 07:34:09 on dist-test-slave-57fn"
server_key: "88c13d5587682ec527dfb8bb1dfdc64c"
server_key_iv: "3765700e08722f0c3a4262cd78775322"
server_key_version: "encryptionkey@0"
I20260925 07:34:09.358868 27239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790321392278483-9480-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790321392278483-9480-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790321392278483-9480-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:34:09.366832 27239 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:34:09.367105 27239 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:34:09.367257 27239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:34:09.367471 27239 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:34:09.367781 27239 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:34:09.367838 27239 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:34:09.367887 27239 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:34:09.367921 27239 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:34:09.373725 27239 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.2:40871
I20260925 07:34:09.373771 27366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.2:40871 every 8 connection(s)
I20260925 07:34:09.374066 27239 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790321392278483-9480-0/minicluster-data/ts-1/data/info.pb
I20260925 07:34:09.376837 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 27239
I20260925 07:34:09.376959 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790321392278483-9480-0/minicluster-data/ts-1/wal/instance
I20260925 07:34:09.377156 9480 external_mini_cluster.cc:1428] Setting key a2eb177fad4204ef0df5929137d7ec66
I20260925 07:34:09.378167 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790321392278483-9480-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790321392278483-9480-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790321392278483-9480-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790321392278483-9480-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.66.3:0
--local_ip_for_outbound_sockets=127.9.66.3
--webserver_interface=127.9.66.3
--webserver_port=0
--tserver_master_addrs=127.9.66.62:34249
--builtin_ntp_servers=127.9.66.20:43747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 07:34:09.379261 27088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:09.374749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.2:37713 (local address 127.9.66.62:34249)
0925 07:34:09.374811 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:09.374813 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:09.375016 (+ 203us) server_negotiation.cc:407] Connection header received
0925 07:34:09.375424 (+ 408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:09.375425 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:09.375435 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:09.375469 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:34:09.376044 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:09.376525 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:09.377942 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:09.378155 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:09.378206 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:09.379031 (+ 825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:34:09.379032 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:34:09.379032 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 07:34:09.379036 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:34:09.379050 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 07:34:09.379053 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:34:09.379090 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:09.379187 (+ 97us) server_negotiation.cc:299] Negotiation successful
0925 07:34:09.379209 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 07:34:09.379582 27369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:09.374737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34249 (local address 127.9.66.2:37713)
0925 07:34:09.374901 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 07:34:09.374936 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 07:34:09.375335 (+ 399us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:34:09.375522 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:34:09.375525 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:34:09.375532 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:34:09.375979 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:09.375983 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:09.376601 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:09.376601 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:09.377852 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:09.377855 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:09.378241 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:09.378242 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:09.378393 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:09.378867 (+ 474us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 07:34:09.378867 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:34:09.378900 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 07:34:09.378928 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:34:09.379109 (+ 181us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:34:09.379114 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:34:09.379116 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 07:34:09.379193 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 07:34:09.379297 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":79,"threads_started":1}
I20260925 07:34:09.380107 27367 heartbeater.cc:347] Connected to a master server at 127.9.66.62:34249
I20260925 07:34:09.380205 27367 heartbeater.cc:464] Registering TS with master...
I20260925 07:34:09.380417 27367 heartbeater.cc:511] Master 127.9.66.62:34249 requested a full tablet report, sending...
I20260925 07:34:09.380808 27048 ts_manager.cc:194] Registered new tserver with Master: 9a8822c17cdb45b5b619c7a2fad59cd6 (127.9.66.2:40871)
I20260925 07:34:09.381335 27048 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.66.2:37713
W20260925 07:34:09.455650 27370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:34:09.455821 27370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:34:09.455842 27370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:34:09.456959 27370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:34:09.456996 27370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:34:09.457048 27370 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.66.3
I20260925 07:34:09.458352 27370 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:43747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790321392278483-9480-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790321392278483-9480-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.66.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790321392278483-9480-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.66.3
--webserver_port=0
--tserver_master_addrs=127.9.66.62:34249
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27370
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.66.3
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790321392278483-9480-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:34:09.458581 27370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:34:09.458845 27370 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:34:09.463492 27375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:34:09.463508 27378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:34:09.463511 27376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:34:09.463726 27370 server_base.cc:1034] running on GCE node
I20260925 07:34:09.463940 27370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:34:09.464171 27370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:34:09.465337 27370 hybrid_clock.cc:648] HybridClock initialized: now 1790321649465313 us; error 37 us; skew 500 ppm
I20260925 07:34:09.466614 27370 webserver.cc:460] Webserver started at http://127.9.66.3:39955/ using document root <none> and password file <none>
I20260925 07:34:09.466852 27370 fs_manager.cc:359] Metadata directory not provided
I20260925 07:34:09.466923 27370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:34:09.467057 27370 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:34:09.468075 27370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790321392278483-9480-0/minicluster-data/ts-2/data/instance:
uuid: "a079b9d327f641b585e03e414ee5d2be"
format_stamp: "Formatted at 2026-09-25 07:34:09 on dist-test-slave-57fn"
server_key: "f72a11c88f1e2ed90461293af3bb9ff0"
server_key_iv: "3facaf2d8520444330b547f8164f9adc"
server_key_version: "encryptionkey@0"
I20260925 07:34:09.468438 27370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790321392278483-9480-0/minicluster-data/ts-2/wal/instance:
uuid: "a079b9d327f641b585e03e414ee5d2be"
format_stamp: "Formatted at 2026-09-25 07:34:09 on dist-test-slave-57fn"
server_key: "f72a11c88f1e2ed90461293af3bb9ff0"
server_key_iv: "3facaf2d8520444330b547f8164f9adc"
server_key_version: "encryptionkey@0"
I20260925 07:34:09.469703 27370 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 07:34:09.470444 27384 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:34:09.470644 27370 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 07:34:09.470739 27370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790321392278483-9480-0/minicluster-data/ts-2/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790321392278483-9480-0/minicluster-data/ts-2/wal
uuid: "a079b9d327f641b585e03e414ee5d2be"
format_stamp: "Formatted at 2026-09-25 07:34:09 on dist-test-slave-57fn"
server_key: "f72a11c88f1e2ed90461293af3bb9ff0"
server_key_iv: "3facaf2d8520444330b547f8164f9adc"
server_key_version: "encryptionkey@0"
I20260925 07:34:09.470813 27370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790321392278483-9480-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790321392278483-9480-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790321392278483-9480-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:34:09.481127 27370 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:34:09.481386 27370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:34:09.481586 27370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:34:09.481796 27370 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:34:09.482102 27370 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:34:09.482155 27370 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:34:09.482189 27370 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:34:09.482216 27370 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:34:09.487661 27370 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.3:42527
I20260925 07:34:09.487709 27497 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.3:42527 every 8 connection(s)
I20260925 07:34:09.488004 27370 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790321392278483-9480-0/minicluster-data/ts-2/data/info.pb
I20260925 07:34:09.492743 27088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:09.488696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.3:49903 (local address 127.9.66.62:34249)
0925 07:34:09.488761 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:09.488763 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:09.488960 (+ 197us) server_negotiation.cc:407] Connection header received
0925 07:34:09.489385 (+ 425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:09.489386 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:09.489396 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:09.489445 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:34:09.490092 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:09.490604 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:09.491564 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:09.491749 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:09.491797 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:09.492523 (+ 726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:34:09.492524 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:34:09.492524 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 07:34:09.492528 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:34:09.492539 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 07:34:09.492543 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:34:09.492583 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:09.492666 (+ 83us) server_negotiation.cc:299] Negotiation successful
0925 07:34:09.492686 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 07:34:09.492959 27500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:09.488671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34249 (local address 127.9.66.3:49903)
0925 07:34:09.488841 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 07:34:09.488872 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0925 07:34:09.489304 (+ 432us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:34:09.489492 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:34:09.489495 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:34:09.489541 (+ 46us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:34:09.490020 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:09.490024 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:09.490694 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:09.490694 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:09.491482 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:09.491483 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:09.491817 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:09.491817 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:09.491922 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:09.492393 (+ 471us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 07:34:09.492394 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:34:09.492424 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 07:34:09.492446 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:34:09.492597 (+ 151us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:34:09.492601 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:34:09.492603 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 07:34:09.492652 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 07:34:09.492726 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":93,"threads_started":1}
I20260925 07:34:09.493575 27498 heartbeater.cc:347] Connected to a master server at 127.9.66.62:34249
I20260925 07:34:09.493587 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 27370
I20260925 07:34:09.493664 27498 heartbeater.cc:464] Registering TS with master...
I20260925 07:34:09.493698 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790321392278483-9480-0/minicluster-data/ts-2/wal/instance
I20260925 07:34:09.493851 27498 heartbeater.cc:511] Master 127.9.66.62:34249 requested a full tablet report, sending...
I20260925 07:34:09.493896 9480 external_mini_cluster.cc:1428] Setting key dd003be2a53404f32e4b0310d991b5da
I20260925 07:34:09.494213 27048 ts_manager.cc:194] Registered new tserver with Master: a079b9d327f641b585e03e414ee5d2be (127.9.66.3:42527)
I20260925 07:34:09.494705 27048 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.66.3:49903
I20260925 07:34:09.494761 9480 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 07:34:09.498998 27088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:09.496168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35476 (local address 127.9.66.62:34249)
0925 07:34:09.496220 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:09.496222 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:09.496339 (+ 117us) server_negotiation.cc:407] Connection header received
0925 07:34:09.496429 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:09.496430 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:09.496442 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:09.496485 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:34:09.496656 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:09.497098 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:09.497946 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:09.498127 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:09.498212 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:09.498773 (+ 561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:34:09.498774 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:34:09.498775 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 07:34:09.498779 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:34:09.498791 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 07:34:09.498795 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:34:09.498849 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:09.498907 (+ 58us) server_negotiation.cc:299] Negotiation successful
0925 07:34:09.498925 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 07:34:09.500481 27048 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35476:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 07:34:09.502252 27048 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 07:34:09.508539 27506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:09.504207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.3:42527 (local address 127.0.0.1:56308)
0925 07:34:09.504502 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0925 07:34:09.504511 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 07:34:09.504548 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:34:09.504774 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:34:09.504776 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:34:09.504777 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:34:09.504820 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:09.504841 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:09.506169 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:09.506170 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:09.507296 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:09.507298 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:09.508253 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:09.508254 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:09.508351 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:09.508366 (+ 15us) client_negotiation.cc:769] Sending connection context
0925 07:34:09.508413 (+ 47us) client_negotiation.cc:240] Negotiation successful
0925 07:34:09.508458 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"mutex_wait_us":94,"thread_start_us":80,"threads_started":1}
I20260925 07:34:09.508795 27509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:09.504391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.1:33225 (local address 127.0.0.1:42386)
0925 07:34:09.504866 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0925 07:34:09.504877 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 07:34:09.504940 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:34:09.505164 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:34:09.505165 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:34:09.505166 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:34:09.505215 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:09.505216 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:09.506127 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:09.506128 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:09.507538 (+ 1410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:09.507541 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:09.508560 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:09.508560 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:09.508621 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:09.508626 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 07:34:09.508677 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 07:34:09.508717 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"mutex_wait_us":126,"thread_start_us":62,"threads_started":1}
I20260925 07:34:09.508888 27510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:09.504445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56308 (local address 127.9.66.3:42527)
0925 07:34:09.504607 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:09.504611 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:09.504623 (+ 12us) server_negotiation.cc:407] Connection header received
0925 07:34:09.504660 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:09.504663 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:09.504677 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:09.504773 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:34:09.504925 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:09.506061 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:09.507375 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:09.508140 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:09.508661 (+ 521us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:09.508748 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:09.508772 (+ 24us) server_negotiation.cc:299] Negotiation successful
0925 07:34:09.508808 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":71,"threads_started":1}
I20260925 07:34:09.509016 27508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:09.504283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58164 (local address 127.9.66.2:40871)
0925 07:34:09.504713 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:09.504716 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:09.504725 (+ 9us) server_negotiation.cc:407] Connection header received
0925 07:34:09.504761 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:09.504763 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:09.504777 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:09.504854 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:34:09.504989 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:09.506119 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:09.507642 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:09.508370 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:09.508438 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:09.508521 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:09.508838 (+ 317us) server_negotiation.cc:299] Negotiation successful
0925 07:34:09.508889 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":72,"threads_started":1}
I20260925 07:34:09.509181 27507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:09.504162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.2:40871 (local address 127.0.0.1:58164)
0925 07:34:09.504356 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0925 07:34:09.504392 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 07:34:09.504452 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:34:09.504863 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:34:09.504864 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:34:09.504866 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:34:09.504902 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:09.504903 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:09.506366 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:09.506366 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:09.507538 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:09.507541 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:09.508558 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:09.508559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:09.508711 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:09.508713 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 07:34:09.508752 (+ 39us) client_negotiation.cc:240] Negotiation successful
0925 07:34:09.508782 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"mutex_wait_us":7,"thread_start_us":83,"threads_started":1}
I20260925 07:34:09.509589 27511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:09.504518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42386 (local address 127.9.66.1:33225)
0925 07:34:09.504670 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:09.504674 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:09.504938 (+ 264us) server_negotiation.cc:407] Connection header received
0925 07:34:09.505009 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:09.505011 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:09.505025 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:09.505095 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:34:09.505291 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:09.506037 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:09.507642 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:09.508366 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:09.508436 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:09.508521 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:09.509218 (+ 697us) server_negotiation.cc:299] Negotiation successful
0925 07:34:09.509495 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":83,"threads_started":1}
I20260925 07:34:09.509627 27432 tablet_service.cc:1467] Processing CreateTablet for tablet ac972c07e1ae4f43b5046b98381470cc (DEFAULT_TABLE table=test-table [id=2d687cc244d141b89f0793e0d1c86f52]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:34:09.509864 27301 tablet_service.cc:1467] Processing CreateTablet for tablet ac972c07e1ae4f43b5046b98381470cc (DEFAULT_TABLE table=test-table [id=2d687cc244d141b89f0793e0d1c86f52]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:34:09.509994 27432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac972c07e1ae4f43b5046b98381470cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:34:09.510169 27301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac972c07e1ae4f43b5046b98381470cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:34:09.510288 27170 tablet_service.cc:1467] Processing CreateTablet for tablet ac972c07e1ae4f43b5046b98381470cc (DEFAULT_TABLE table=test-table [id=2d687cc244d141b89f0793e0d1c86f52]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:34:09.510574 27170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac972c07e1ae4f43b5046b98381470cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:34:09.512311 27512 tablet_bootstrap.cc:492] T ac972c07e1ae4f43b5046b98381470cc P a079b9d327f641b585e03e414ee5d2be: Bootstrap starting.
I20260925 07:34:09.513115 27512 tablet_bootstrap.cc:654] T ac972c07e1ae4f43b5046b98381470cc P a079b9d327f641b585e03e414ee5d2be: Neither blocks nor log segments found. Creating new log.
I20260925 07:34:09.513411 27512 log.cc:824] T ac972c07e1ae4f43b5046b98381470cc P a079b9d327f641b585e03e414ee5d2be: Log is configured to *not* fsync() on all Append() calls
I20260925 07:34:09.514104 27515 tablet_bootstrap.cc:492] T ac972c07e1ae4f43b5046b98381470cc P 2d71fd955d5044c1b3dae2fc6f54d59e: Bootstrap starting.
I20260925 07:34:09.514209 27512 tablet_bootstrap.cc:492] T ac972c07e1ae4f43b5046b98381470cc P a079b9d327f641b585e03e414ee5d2be: No bootstrap required, opened a new log
I20260925 07:34:09.514250 27513 tablet_bootstrap.cc:492] T ac972c07e1ae4f43b5046b98381470cc P 9a8822c17cdb45b5b619c7a2fad59cd6: Bootstrap starting.
I20260925 07:34:09.514321 27512 ts_tablet_manager.cc:1397] T ac972c07e1ae4f43b5046b98381470cc P a079b9d327f641b585e03e414ee5d2be: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 07:34:09.514879 27513 tablet_bootstrap.cc:654] T ac972c07e1ae4f43b5046b98381470cc P 9a8822c17cdb45b5b619c7a2fad59cd6: Neither blocks nor log segments found. Creating new log.
I20260925 07:34:09.514889 27515 tablet_bootstrap.cc:654] T ac972c07e1ae4f43b5046b98381470cc P 2d71fd955d5044c1b3dae2fc6f54d59e: Neither blocks nor log segments found. Creating new log.
I20260925 07:34:09.515206 27515 log.cc:824] T ac972c07e1ae4f43b5046b98381470cc P 2d71fd955d5044c1b3dae2fc6f54d59e: Log is configured to *not* fsync() on all Append() calls
I20260925 07:34:09.515206 27513 log.cc:824] T ac972c07e1ae4f43b5046b98381470cc P 9a8822c17cdb45b5b619c7a2fad59cd6: Log is configured to *not* fsync() on all Append() calls
I20260925 07:34:09.515945 27513 tablet_bootstrap.cc:492] T ac972c07e1ae4f43b5046b98381470cc P 9a8822c17cdb45b5b619c7a2fad59cd6: No bootstrap required, opened a new log
I20260925 07:34:09.516062 27513 ts_tablet_manager.cc:1397] T ac972c07e1ae4f43b5046b98381470cc P 9a8822c17cdb45b5b619c7a2fad59cd6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 07:34:09.516295 27515 tablet_bootstrap.cc:492] T ac972c07e1ae4f43b5046b98381470cc P 2d71fd955d5044c1b3dae2fc6f54d59e: No bootstrap required, opened a new log
I20260925 07:34:09.516402 27515 ts_tablet_manager.cc:1397] T ac972c07e1ae4f43b5046b98381470cc P 2d71fd955d5044c1b3dae2fc6f54d59e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 07:34:09.516639 27512 raft_consensus.cc:348] T ac972c07e1ae4f43b5046b98381470cc P a079b9d327f641b585e03e414ee5d2be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a8822c17cdb45b5b619c7a2fad59cd6" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 40871 } } peers { permanent_uuid: "a079b9d327f641b585e03e414ee5d2be" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 42527 } } peers { permanent_uuid: "2d71fd955d5044c1b3dae2fc6f54d59e" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 33225 } }
I20260925 07:34:09.516788 27512 raft_consensus.cc:374] T ac972c07e1ae4f43b5046b98381470cc P a079b9d327f641b585e03e414ee5d2be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:34:09.516850 27512 raft_consensus.cc:729] T ac972c07e1ae4f43b5046b98381470cc P a079b9d327f641b585e03e414ee5d2be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a079b9d327f641b585e03e414ee5d2be, State: Initialized, Role: FOLLOWER
I20260925 07:34:09.516976 27512 consensus_queue.cc:260] T ac972c07e1ae4f43b5046b98381470cc P a079b9d327f641b585e03e414ee5d2be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a8822c17cdb45b5b619c7a2fad59cd6" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 40871 } } peers { permanent_uuid: "a079b9d327f641b585e03e414ee5d2be" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 42527 } } peers { permanent_uuid: "2d71fd955d5044c1b3dae2fc6f54d59e" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 33225 } }
I20260925 07:34:09.517241 27512 ts_tablet_manager.cc:1428] T ac972c07e1ae4f43b5046b98381470cc P a079b9d327f641b585e03e414ee5d2be: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 07:34:09.517364 27498 heartbeater.cc:503] Master 127.9.66.62:34249 was elected leader, sending a full tablet report...
I20260925 07:34:09.518787 27513 raft_consensus.cc:348] T ac972c07e1ae4f43b5046b98381470cc P 9a8822c17cdb45b5b619c7a2fad59cd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a8822c17cdb45b5b619c7a2fad59cd6" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 40871 } } peers { permanent_uuid: "a079b9d327f641b585e03e414ee5d2be" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 42527 } } peers { permanent_uuid: "2d71fd955d5044c1b3dae2fc6f54d59e" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 33225 } }
I20260925 07:34:09.518955 27513 raft_consensus.cc:374] T ac972c07e1ae4f43b5046b98381470cc P 9a8822c17cdb45b5b619c7a2fad59cd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:34:09.518994 27513 raft_consensus.cc:729] T ac972c07e1ae4f43b5046b98381470cc P 9a8822c17cdb45b5b619c7a2fad59cd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a8822c17cdb45b5b619c7a2fad59cd6, State: Initialized, Role: FOLLOWER
I20260925 07:34:09.518950 27515 raft_consensus.cc:348] T ac972c07e1ae4f43b5046b98381470cc P 2d71fd955d5044c1b3dae2fc6f54d59e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a8822c17cdb45b5b619c7a2fad59cd6" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 40871 } } peers { permanent_uuid: "a079b9d327f641b585e03e414ee5d2be" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 42527 } } peers { permanent_uuid: "2d71fd955d5044c1b3dae2fc6f54d59e" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 33225 } }
I20260925 07:34:09.519088 27515 raft_consensus.cc:374] T ac972c07e1ae4f43b5046b98381470cc P 2d71fd955d5044c1b3dae2fc6f54d59e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:34:09.519127 27515 raft_consensus.cc:729] T ac972c07e1ae4f43b5046b98381470cc P 2d71fd955d5044c1b3dae2fc6f54d59e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d71fd955d5044c1b3dae2fc6f54d59e, State: Initialized, Role: FOLLOWER
I20260925 07:34:09.519110 27513 consensus_queue.cc:260] T ac972c07e1ae4f43b5046b98381470cc P 9a8822c17cdb45b5b619c7a2fad59cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a8822c17cdb45b5b619c7a2fad59cd6" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 40871 } } peers { permanent_uuid: "a079b9d327f641b585e03e414ee5d2be" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 42527 } } peers { permanent_uuid: "2d71fd955d5044c1b3dae2fc6f54d59e" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 33225 } }
I20260925 07:34:09.519232 27515 consensus_queue.cc:260] T ac972c07e1ae4f43b5046b98381470cc P 2d71fd955d5044c1b3dae2fc6f54d59e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a8822c17cdb45b5b619c7a2fad59cd6" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 40871 } } peers { permanent_uuid: "a079b9d327f641b585e03e414ee5d2be" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 42527 } } peers { permanent_uuid: "2d71fd955d5044c1b3dae2fc6f54d59e" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 33225 } }
I20260925 07:34:09.519363 27513 ts_tablet_manager.cc:1428] T ac972c07e1ae4f43b5046b98381470cc P 9a8822c17cdb45b5b619c7a2fad59cd6: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 07:34:09.519388 27367 heartbeater.cc:503] Master 127.9.66.62:34249 was elected leader, sending a full tablet report...
I20260925 07:34:09.519505 27515 ts_tablet_manager.cc:1428] T ac972c07e1ae4f43b5046b98381470cc P 2d71fd955d5044c1b3dae2fc6f54d59e: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 07:34:09.519626 27236 heartbeater.cc:503] Master 127.9.66.62:34249 was elected leader, sending a full tablet report...
W20260925 07:34:09.624624 27368 tablet.cc:2369] T ac972c07e1ae4f43b5046b98381470cc P 9a8822c17cdb45b5b619c7a2fad59cd6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 07:34:09.676009 27520 raft_consensus.cc:482] T ac972c07e1ae4f43b5046b98381470cc P 2d71fd955d5044c1b3dae2fc6f54d59e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 07:34:09.676121 27520 raft_consensus.cc:504] T ac972c07e1ae4f43b5046b98381470cc P 2d71fd955d5044c1b3dae2fc6f54d59e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a8822c17cdb45b5b619c7a2fad59cd6" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 40871 } } peers { permanent_uuid: "a079b9d327f641b585e03e414ee5d2be" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 42527 } } peers { permanent_uuid: "2d71fd955d5044c1b3dae2fc6f54d59e" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 33225 } }
I20260925 07:34:09.676623 27520 leader_election.cc:288] T ac972c07e1ae4f43b5046b98381470cc P 2d71fd955d5044c1b3dae2fc6f54d59e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a8822c17cdb45b5b619c7a2fad59cd6 (127.9.66.2:40871), a079b9d327f641b585e03e414ee5d2be (127.9.66.3:42527)
I20260925 07:34:09.679592 27238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:09.676815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.3:42527 (local address 127.9.66.1:42523)
0925 07:34:09.676876 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0925 07:34:09.676903 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0925 07:34:09.676948 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:34:09.677117 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:34:09.677118 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:34:09.677120 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:34:09.677200 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:09.677202 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:09.677998 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:09.677999 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:09.678744 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:09.678746 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:09.679329 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:09.679330 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:09.679402 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:09.679405 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 07:34:09.679457 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 07:34:09.679496 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260925 07:34:09.679605 27510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:09.676888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.1:42523 (local address 127.9.66.3:42527)
0925 07:34:09.676946 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:09.676948 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:09.676962 (+ 14us) server_negotiation.cc:407] Connection header received
0925 07:34:09.677021 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:09.677022 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:09.677035 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:09.677099 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:34:09.677279 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:09.677918 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:09.678839 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:09.679237 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:09.679294 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:09.679302 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:09.679481 (+ 179us) server_negotiation.cc:299] Negotiation successful
0925 07:34:09.679529 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260925 07:34:09.679672 27508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:09.676958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.1:43357 (local address 127.9.66.2:40871)
0925 07:34:09.677002 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:09.677004 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:09.677104 (+ 100us) server_negotiation.cc:407] Connection header received
0925 07:34:09.677158 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:09.677159 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:09.677169 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:09.677239 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:34:09.677444 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:09.678008 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:09.678959 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:09.679320 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:09.679412 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:09.679422 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:09.679566 (+ 144us) server_negotiation.cc:299] Negotiation successful
0925 07:34:09.679601 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 07:34:09.679729 27521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:09.676825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.2:40871 (local address 127.9.66.1:43357)
0925 07:34:09.677017 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0925 07:34:09.677028 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 07:34:09.677090 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:34:09.677264 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:34:09.677265 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:34:09.677267 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:34:09.677341 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:09.677343 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:09.678085 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:09.678086 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:09.678867 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:09.678869 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:09.679398 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:09.679399 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:09.679488 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:09.679493 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 07:34:09.679554 (+ 61us) client_negotiation.cc:240] Negotiation successful
0925 07:34:09.679592 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"mutex_wait_us":4,"thread_start_us":101,"threads_started":1}
I20260925 07:34:09.679943 27452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac972c07e1ae4f43b5046b98381470cc" candidate_uuid: "2d71fd955d5044c1b3dae2fc6f54d59e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a079b9d327f641b585e03e414ee5d2be" is_pre_election: true
I20260925 07:34:09.680034 27321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac972c07e1ae4f43b5046b98381470cc" candidate_uuid: "2d71fd955d5044c1b3dae2fc6f54d59e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a8822c17cdb45b5b619c7a2fad59cd6" is_pre_election: true
I20260925 07:34:09.680089 27452 raft_consensus.cc:2454] T ac972c07e1ae4f43b5046b98381470cc P a079b9d327f641b585e03e414ee5d2be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d71fd955d5044c1b3dae2fc6f54d59e in term 0.
I20260925 07:34:09.680188 27321 raft_consensus.cc:2454] T ac972c07e1ae4f43b5046b98381470cc P 9a8822c17cdb45b5b619c7a2fad59cd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d71fd955d5044c1b3dae2fc6f54d59e in term 0.
I20260925 07:34:09.680325 27125 leader_election.cc:302] T ac972c07e1ae4f43b5046b98381470cc P 2d71fd955d5044c1b3dae2fc6f54d59e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d71fd955d5044c1b3dae2fc6f54d59e, a079b9d327f641b585e03e414ee5d2be; no voters:
I20260925 07:34:09.680433 27520 raft_consensus.cc:2781] T ac972c07e1ae4f43b5046b98381470cc P 2d71fd955d5044c1b3dae2fc6f54d59e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 07:34:09.680481 27520 raft_consensus.cc:482] T ac972c07e1ae4f43b5046b98381470cc P 2d71fd955d5044c1b3dae2fc6f54d59e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 07:34:09.680533 27520 raft_consensus.cc:3037] T ac972c07e1ae4f43b5046b98381470cc P 2d71fd955d5044c1b3dae2fc6f54d59e [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:34:09.681262 27520 raft_consensus.cc:504] T ac972c07e1ae4f43b5046b98381470cc P 2d71fd955d5044c1b3dae2fc6f54d59e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a8822c17cdb45b5b619c7a2fad59cd6" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 40871 } } peers { permanent_uuid: "a079b9d327f641b585e03e414ee5d2be" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 42527 } } peers { permanent_uuid: "2d71fd955d5044c1b3dae2fc6f54d59e" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 33225 } }
I20260925 07:34:09.681396 27520 leader_election.cc:288] T ac972c07e1ae4f43b5046b98381470cc P 2d71fd955d5044c1b3dae2fc6f54d59e [CANDIDATE]: Term 1 election: Requested vote from peers 9a8822c17cdb45b5b619c7a2fad59cd6 (127.9.66.2:40871), a079b9d327f641b585e03e414ee5d2be (127.9.66.3:42527)
I20260925 07:34:09.681540 27321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac972c07e1ae4f43b5046b98381470cc" candidate_uuid: "2d71fd955d5044c1b3dae2fc6f54d59e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a8822c17cdb45b5b619c7a2fad59cd6"
I20260925 07:34:09.681552 27452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac972c07e1ae4f43b5046b98381470cc" candidate_uuid: "2d71fd955d5044c1b3dae2fc6f54d59e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a079b9d327f641b585e03e414ee5d2be"
I20260925 07:34:09.681623 27321 raft_consensus.cc:3037] T ac972c07e1ae4f43b5046b98381470cc P 9a8822c17cdb45b5b619c7a2fad59cd6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:34:09.681645 27452 raft_consensus.cc:3037] T ac972c07e1ae4f43b5046b98381470cc P a079b9d327f641b585e03e414ee5d2be [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:34:09.682294 27452 raft_consensus.cc:2454] T ac972c07e1ae4f43b5046b98381470cc P a079b9d327f641b585e03e414ee5d2be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d71fd955d5044c1b3dae2fc6f54d59e in term 1.
I20260925 07:34:09.682338 27321 raft_consensus.cc:2454] T ac972c07e1ae4f43b5046b98381470cc P 9a8822c17cdb45b5b619c7a2fad59cd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d71fd955d5044c1b3dae2fc6f54d59e in term 1.
I20260925 07:34:09.682495 27124 leader_election.cc:302] T ac972c07e1ae4f43b5046b98381470cc P 2d71fd955d5044c1b3dae2fc6f54d59e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d71fd955d5044c1b3dae2fc6f54d59e, 9a8822c17cdb45b5b619c7a2fad59cd6; no voters:
I20260925 07:34:09.682611 27520 raft_consensus.cc:2781] T ac972c07e1ae4f43b5046b98381470cc P 2d71fd955d5044c1b3dae2fc6f54d59e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:34:09.682749 27520 raft_consensus.cc:686] T ac972c07e1ae4f43b5046b98381470cc P 2d71fd955d5044c1b3dae2fc6f54d59e [term 1 LEADER]: Becoming Leader. State: Replica: 2d71fd955d5044c1b3dae2fc6f54d59e, State: Running, Role: LEADER
I20260925 07:34:09.682816 27520 consensus_queue.cc:237] T ac972c07e1ae4f43b5046b98381470cc P 2d71fd955d5044c1b3dae2fc6f54d59e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a8822c17cdb45b5b619c7a2fad59cd6" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 40871 } } peers { permanent_uuid: "a079b9d327f641b585e03e414ee5d2be" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 42527 } } peers { permanent_uuid: "2d71fd955d5044c1b3dae2fc6f54d59e" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 33225 } }
I20260925 07:34:09.683703 27047 catalog_manager.cc:5495] T ac972c07e1ae4f43b5046b98381470cc P 2d71fd955d5044c1b3dae2fc6f54d59e reported cstate change: term changed from 0 to 1, leader changed from <none> to 2d71fd955d5044c1b3dae2fc6f54d59e (127.9.66.1). New cstate: current_term: 1 leader_uuid: "2d71fd955d5044c1b3dae2fc6f54d59e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a8822c17cdb45b5b619c7a2fad59cd6" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 40871 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a079b9d327f641b585e03e414ee5d2be" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 42527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d71fd955d5044c1b3dae2fc6f54d59e" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 33225 } health_report { overall_health: HEALTHY } } }
I20260925 07:34:09.688019 27511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:09.686098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42394 (local address 127.9.66.1:33225)
0925 07:34:09.686155 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:09.686157 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:09.686165 (+ 8us) server_negotiation.cc:407] Connection header received
0925 07:34:09.686193 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:09.686193 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:09.686222 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:09.686282 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 07:34:09.686372 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:09.686781 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:09.687446 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:09.687617 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:09.687670 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:09.687786 (+ 116us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 07:34:09.687863 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 07:34:09.687929 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:09.687946 (+ 17us) server_negotiation.cc:299] Negotiation successful
0925 07:34:09.687961 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 07:34:09.689483 27452 raft_consensus.cc:1261] T ac972c07e1ae4f43b5046b98381470cc P a079b9d327f641b585e03e414ee5d2be [term 1 FOLLOWER]: Refusing update from remote peer 2d71fd955d5044c1b3dae2fc6f54d59e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:34:09.689594 27321 raft_consensus.cc:1261] T ac972c07e1ae4f43b5046b98381470cc P 9a8822c17cdb45b5b619c7a2fad59cd6 [term 1 FOLLOWER]: Refusing update from remote peer 2d71fd955d5044c1b3dae2fc6f54d59e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:34:09.689950 27520 consensus_queue.cc:1035] T ac972c07e1ae4f43b5046b98381470cc P 2d71fd955d5044c1b3dae2fc6f54d59e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a8822c17cdb45b5b619c7a2fad59cd6" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 40871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:34:09.690083 27522 consensus_queue.cc:1035] T ac972c07e1ae4f43b5046b98381470cc P 2d71fd955d5044c1b3dae2fc6f54d59e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a079b9d327f641b585e03e414ee5d2be" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 42527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:34:09.692358 27525 mvcc.cc:205] Tried to move back new op lower bound from 7333157477125939200 to 7333157477101666304. Current Snapshot: MvccSnapshot[applied={T|T < 7333157477125939200}]
I20260925 07:34:09.692734 27526 mvcc.cc:205] Tried to move back new op lower bound from 7333157477125939200 to 7333157477101666304. Current Snapshot: MvccSnapshot[applied={T|T < 7333157477125939200}]
I20260925 07:34:09.693111 27527 mvcc.cc:205] Tried to move back new op lower bound from 7333157477125939200 to 7333157477101666304. Current Snapshot: MvccSnapshot[applied={T|T < 7333157477125939200}]
I20260925 07:34:09.694998 27046 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35476:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 07:34:09.695079 27046 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35476:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 07:34:09.695755 27046 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 03d9b59ddca741b888127e072d46371d: Sending DeleteTablet for 3 replicas of tablet ac972c07e1ae4f43b5046b98381470cc
I20260925 07:34:09.696132 27432 tablet_service.cc:1514] Processing DeleteTablet for tablet ac972c07e1ae4f43b5046b98381470cc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:34:09 UTC) from {username='slave'} at 127.0.0.1:56308
I20260925 07:34:09.696141 27170 tablet_service.cc:1514] Processing DeleteTablet for tablet ac972c07e1ae4f43b5046b98381470cc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:34:09 UTC) from {username='slave'} at 127.0.0.1:42386
I20260925 07:34:09.696354 27301 tablet_service.cc:1514] Processing DeleteTablet for tablet ac972c07e1ae4f43b5046b98381470cc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:34:09 UTC) from {username='slave'} at 127.0.0.1:58164
I20260925 07:34:09.696569 27535 tablet_replica.cc:330] T ac972c07e1ae4f43b5046b98381470cc P a079b9d327f641b585e03e414ee5d2be: stopping tablet replica
I20260925 07:34:09.696575 27536 tablet_replica.cc:330] T ac972c07e1ae4f43b5046b98381470cc P 9a8822c17cdb45b5b619c7a2fad59cd6: stopping tablet replica
I20260925 07:34:09.696659 27535 raft_consensus.cc:2229] T ac972c07e1ae4f43b5046b98381470cc P a079b9d327f641b585e03e414ee5d2be [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 07:34:09.696660 27536 raft_consensus.cc:2229] T ac972c07e1ae4f43b5046b98381470cc P 9a8822c17cdb45b5b619c7a2fad59cd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 07:34:09.696753 27536 raft_consensus.cc:2258] T ac972c07e1ae4f43b5046b98381470cc P 9a8822c17cdb45b5b619c7a2fad59cd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:34:09.696756 27535 raft_consensus.cc:2258] T ac972c07e1ae4f43b5046b98381470cc P a079b9d327f641b585e03e414ee5d2be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:34:09.696825 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 27108
I20260925 07:34:09.697793 27535 ts_tablet_manager.cc:1905] T ac972c07e1ae4f43b5046b98381470cc P a079b9d327f641b585e03e414ee5d2be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 07:34:09.699394 27535 ts_tablet_manager.cc:1918] T ac972c07e1ae4f43b5046b98381470cc P a079b9d327f641b585e03e414ee5d2be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 07:34:09.699476 27535 log.cc:1196] T ac972c07e1ae4f43b5046b98381470cc P a079b9d327f641b585e03e414ee5d2be: Deleting WAL directory at /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790321392278483-9480-0/minicluster-data/ts-2/wal/wals/ac972c07e1ae4f43b5046b98381470cc
I20260925 07:34:09.699700 27535 ts_tablet_manager.cc:1939] T ac972c07e1ae4f43b5046b98381470cc P a079b9d327f641b585e03e414ee5d2be: Deleting consensus metadata
I20260925 07:34:09.700084 27034 catalog_manager.cc:4841] TS a079b9d327f641b585e03e414ee5d2be (127.9.66.3:42527): tablet ac972c07e1ae4f43b5046b98381470cc (table test-table [id=2d687cc244d141b89f0793e0d1c86f52]) successfully deleted
I20260925 07:34:09.702356 27536 ts_tablet_manager.cc:1905] T ac972c07e1ae4f43b5046b98381470cc P 9a8822c17cdb45b5b619c7a2fad59cd6: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 07:34:09.702442 27033 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.66.1:33225 (error 108)
I20260925 07:34:09.702879 27507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:09.702769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.1:33225 (local address 127.0.0.1:42404)
0925 07:34:09.702807 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0925 07:34:09.702845 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.1:33225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260925 07:34:09.702981 27033 catalog_manager.cc:4584] TS 2d71fd955d5044c1b3dae2fc6f54d59e (127.9.66.1:33225): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ac972c07e1ae4f43b5046b98381470cc: Network error: Client connection negotiation failed: client connection to 127.9.66.1:33225: connect: Connection refused (error 111)
I20260925 07:34:09.702991 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 27239
I20260925 07:34:09.707628 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 27370
I20260925 07:34:09.707680 27507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:09.707575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.2:40871 (local address 127.0.0.1:58168)
0925 07:34:09.707636 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0925 07:34:09.707660 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.2:40871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260925 07:34:09.712203 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 27017
2026-09-25T07:34:09Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (731 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790321392278483-9480-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 07:34:09 dist-test-slave-57fn krb5kdc[27540](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 07:34:09 dist-test-slave-57fn krb5kdc[27540](info): set up 2 sockets
Sep 25 07:34:09 dist-test-slave-57fn krb5kdc[27540](info): commencing operation
krb5kdc: starting...
W20260925 07:34:10.765345 9480 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790321392278483-9480-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790321392278483-9480-0/krb5kdc/test-user.keytab.
Sep 25 07:34:10 dist-test-slave-57fn krb5kdc[27540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321650, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T07:34:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T07:34:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.66.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.66.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.66.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab.
Entry for principal kudu/127.9.66.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab.
I20260925 07:34:10.884294 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790321392278483-9480-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790321392278483-9480-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790321392278483-9480-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.66.62:42283
--webserver_interface=127.9.66.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab
--principal=kudu/127.9.66.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:36687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.66.62:42283
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:34:10.961005 27553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:34:10.961237 27553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:34:10.961272 27553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:34:10.962484 27553 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:34:10.962553 27553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:34:10.962575 27553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:34:10.962603 27553 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:34:10.962633 27553 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:34:10.964100 27553 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:36687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790321392278483-9480-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790321392278483-9480-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.66.62:42283
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab
--principal=kudu/127.9.66.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.66.62:42283
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.66.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27553
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790321392278483-9480-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:34:10.964366 27553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:34:10.964577 27553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:34:10.969141 27561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:34:10.969280 27559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:34:10.969151 27558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:34:10.969453 27553 server_base.cc:1034] running on GCE node
I20260925 07:34:10.969698 27553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:34:10.969977 27553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:34:10.971150 27553 hybrid_clock.cc:648] HybridClock initialized: now 1790321650971132 us; error 28 us; skew 500 ppm
Sep 25 07:34:10 dist-test-slave-57fn krb5kdc[27540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321650, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:34:10.973492 27553 init.cc:377] Logged in from keytab as kudu/127.9.66.62@KRBTEST.COM (short username kudu)
I20260925 07:34:10.973902 27553 webserver.cc:460] Webserver started at http://127.9.66.62:34415/ using document root <none> and password file <none>
I20260925 07:34:10.974083 27553 fs_manager.cc:359] Metadata directory not provided
I20260925 07:34:10.974121 27553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:34:10.974215 27553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:34:10.975137 27553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790321392278483-9480-0/minicluster-data/master-0/data/instance:
uuid: "6bcfcc1d2cf640efbfbab1bc2633a576"
format_stamp: "Formatted at 2026-09-25 07:34:10 on dist-test-slave-57fn"
server_key: "7f3c24daa1f3ddcc0a928858bc96e5cf"
server_key_iv: "5c7bde38123e9ed5f9293184b6c44c68"
server_key_version: "encryptionkey@0"
I20260925 07:34:10.975442 27553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790321392278483-9480-0/minicluster-data/master-0/wal/instance:
uuid: "6bcfcc1d2cf640efbfbab1bc2633a576"
format_stamp: "Formatted at 2026-09-25 07:34:10 on dist-test-slave-57fn"
server_key: "7f3c24daa1f3ddcc0a928858bc96e5cf"
server_key_iv: "5c7bde38123e9ed5f9293184b6c44c68"
server_key_version: "encryptionkey@0"
I20260925 07:34:10.976668 27553 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 07:34:10.977391 27568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:34:10.977571 27553 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 07:34:10.977660 27553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790321392278483-9480-0/minicluster-data/master-0/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790321392278483-9480-0/minicluster-data/master-0/wal
uuid: "6bcfcc1d2cf640efbfbab1bc2633a576"
format_stamp: "Formatted at 2026-09-25 07:34:10 on dist-test-slave-57fn"
server_key: "7f3c24daa1f3ddcc0a928858bc96e5cf"
server_key_iv: "5c7bde38123e9ed5f9293184b6c44c68"
server_key_version: "encryptionkey@0"
I20260925 07:34:10.977715 27553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790321392278483-9480-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790321392278483-9480-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790321392278483-9480-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:34:10.995988 27553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:34:10.998276 27553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:34:10.998397 27553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:34:11.002286 27553 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.62:42283
I20260925 07:34:11.002394 27620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.62:42283 every 8 connection(s)
I20260925 07:34:11.002697 27553 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
I20260925 07:34:11.003619 27621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:34:11.005927 27621 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bcfcc1d2cf640efbfbab1bc2633a576: Bootstrap starting.
I20260925 07:34:11.006510 27621 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6bcfcc1d2cf640efbfbab1bc2633a576: Neither blocks nor log segments found. Creating new log.
I20260925 07:34:11.006798 27621 log.cc:824] T 00000000000000000000000000000000 P 6bcfcc1d2cf640efbfbab1bc2633a576: Log is configured to *not* fsync() on all Append() calls
I20260925 07:34:11.007517 27621 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bcfcc1d2cf640efbfbab1bc2633a576: No bootstrap required, opened a new log
I20260925 07:34:11.008786 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 27553
I20260925 07:34:11.008906 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790321392278483-9480-0/minicluster-data/master-0/wal/instance
I20260925 07:34:11.009102 9480 external_mini_cluster.cc:1428] Setting key 55160ef08bd9f7e620b8a27296bccfe5
I20260925 07:34:11.009434 27621 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6bcfcc1d2cf640efbfbab1bc2633a576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bcfcc1d2cf640efbfbab1bc2633a576" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 42283 } }
I20260925 07:34:11.009680 27621 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6bcfcc1d2cf640efbfbab1bc2633a576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:34:11.009757 27621 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6bcfcc1d2cf640efbfbab1bc2633a576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bcfcc1d2cf640efbfbab1bc2633a576, State: Initialized, Role: FOLLOWER
I20260925 07:34:11.009910 27621 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6bcfcc1d2cf640efbfbab1bc2633a576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bcfcc1d2cf640efbfbab1bc2633a576" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 42283 } }
I20260925 07:34:11.010005 27621 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6bcfcc1d2cf640efbfbab1bc2633a576 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 07:34:11.010051 27621 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6bcfcc1d2cf640efbfbab1bc2633a576 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 07:34:11.010098 27621 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6bcfcc1d2cf640efbfbab1bc2633a576 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:34:11.011251 27621 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6bcfcc1d2cf640efbfbab1bc2633a576 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bcfcc1d2cf640efbfbab1bc2633a576" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 42283 } }
I20260925 07:34:11.011381 27621 leader_election.cc:302] T 00000000000000000000000000000000 P 6bcfcc1d2cf640efbfbab1bc2633a576 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6bcfcc1d2cf640efbfbab1bc2633a576; no voters:
I20260925 07:34:11.011571 27621 leader_election.cc:288] T 00000000000000000000000000000000 P 6bcfcc1d2cf640efbfbab1bc2633a576 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 07:34:11.011602 27626 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6bcfcc1d2cf640efbfbab1bc2633a576 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:34:11.011778 27626 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6bcfcc1d2cf640efbfbab1bc2633a576 [term 1 LEADER]: Becoming Leader. State: Replica: 6bcfcc1d2cf640efbfbab1bc2633a576, State: Running, Role: LEADER
I20260925 07:34:11.011850 27621 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6bcfcc1d2cf640efbfbab1bc2633a576 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:34:11.011883 27626 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6bcfcc1d2cf640efbfbab1bc2633a576 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bcfcc1d2cf640efbfbab1bc2633a576" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 42283 } }
I20260925 07:34:11.012218 27626 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bcfcc1d2cf640efbfbab1bc2633a576 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6bcfcc1d2cf640efbfbab1bc2633a576. Latest consensus state: current_term: 1 leader_uuid: "6bcfcc1d2cf640efbfbab1bc2633a576" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bcfcc1d2cf640efbfbab1bc2633a576" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 42283 } } }
I20260925 07:34:11.012245 27628 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bcfcc1d2cf640efbfbab1bc2633a576 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6bcfcc1d2cf640efbfbab1bc2633a576" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bcfcc1d2cf640efbfbab1bc2633a576" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 42283 } } }
I20260925 07:34:11.012408 27626 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bcfcc1d2cf640efbfbab1bc2633a576 [sys.catalog]: This master's current role is: LEADER
I20260925 07:34:11.012421 27628 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bcfcc1d2cf640efbfbab1bc2633a576 [sys.catalog]: This master's current role is: LEADER
W20260925 07:34:11.013829 27642 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6bcfcc1d2cf640efbfbab1bc2633a576: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 07:34:11.013883 27642 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 07:34:11.013963 27634 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 07:34:11.014151 27634 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 07:34:11.015786 27634 catalog_manager.cc:1347] Generated new cluster ID: 25fc8d3f764d420e97d3ed78fce0de34
I20260925 07:34:11.015841 27634 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 07:34:11 dist-test-slave-57fn krb5kdc[27540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321650, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.66.62@KRBTEST.COM
I20260925 07:34:11.019876 27625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:11.009707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39828 (local address 127.9.66.62:42283)
0925 07:34:11.009939 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:11.009947 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:11.009971 (+ 24us) server_negotiation.cc:407] Connection header received
0925 07:34:11.010088 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:11.010096 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:11.010150 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:11.010270 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:34:11.011120 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:11.012449 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:11.013433 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:11.014053 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:11.014159 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:11.016709 (+ 2550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:34:11.016713 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:34:11.016714 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:34:11.016725 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:34:11.018338 (+ 1613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:34:11.018706 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:34:11.018709 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:34:11.018709 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:34:11.018810 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:34:11.019046 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:34:11.019047 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:34:11.019047 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:34:11.019278 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:34:11.019474 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:11.019539 (+ 65us) server_negotiation.cc:299] Negotiation successful
0925 07:34:11.019675 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":107,"threads_started":1}
I20260925 07:34:11.050772 27634 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 07:34:11.051442 27634 catalog_manager.cc:1504] Loading token signing keys...
I20260925 07:34:11.056565 27634 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6bcfcc1d2cf640efbfbab1bc2633a576: Generated new TSK 0
I20260925 07:34:11.056767 27634 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.66.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.66.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.66.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.1.keytab.
Entry for principal kudu/127.9.66.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.1.keytab.
I20260925 07:34:11.096007 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790321392278483-9480-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790321392278483-9480-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790321392278483-9480-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790321392278483-9480-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.66.1:0
--local_ip_for_outbound_sockets=127.9.66.1
--webserver_interface=127.9.66.1
--webserver_port=0
--tserver_master_addrs=127.9.66.62:42283
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.1.keytab
--principal=kudu/127.9.66.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:36687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:34:11.206179 27647 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:34:11.206374 27647 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:34:11.206399 27647 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:34:11.207491 27647 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:34:11.207525 27647 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:34:11.207551 27647 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.66.1
I20260925 07:34:11.208884 27647 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:36687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790321392278483-9480-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790321392278483-9480-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.1.keytab
--principal=kudu/127.9.66.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.66.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790321392278483-9480-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.66.1
--webserver_port=0
--tserver_master_addrs=127.9.66.62:42283
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27647
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.66.1
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790321392278483-9480-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:34:11.209059 27647 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:34:11.209233 27647 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:34:11.214182 27653 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:34:11.214234 27655 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:34:11.214226 27652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:34:11.214655 27647 server_base.cc:1034] running on GCE node
I20260925 07:34:11.214850 27647 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:34:11.215143 27647 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:34:11.216337 27647 hybrid_clock.cc:648] HybridClock initialized: now 1790321651216304 us; error 46 us; skew 500 ppm
Sep 25 07:34:11 dist-test-slave-57fn krb5kdc[27540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321651, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:34:11.218992 27647 init.cc:377] Logged in from keytab as kudu/127.9.66.1@KRBTEST.COM (short username kudu)
I20260925 07:34:11.219614 27647 webserver.cc:460] Webserver started at http://127.9.66.1:46439/ using document root <none> and password file <none>
I20260925 07:34:11.219959 27647 fs_manager.cc:359] Metadata directory not provided
I20260925 07:34:11.220052 27647 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:34:11.220196 27647 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:34:11.221800 27647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790321392278483-9480-0/minicluster-data/ts-0/data/instance:
uuid: "0400dbd9ade1475b9078db59ef3ff5c2"
format_stamp: "Formatted at 2026-09-25 07:34:11 on dist-test-slave-57fn"
server_key: "819448bbd27c526dc767d32013f577c6"
server_key_iv: "0c10501b5715d14773f24c693d9e2f0c"
server_key_version: "encryptionkey@0"
I20260925 07:34:11.222167 27647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790321392278483-9480-0/minicluster-data/ts-0/wal/instance:
uuid: "0400dbd9ade1475b9078db59ef3ff5c2"
format_stamp: "Formatted at 2026-09-25 07:34:11 on dist-test-slave-57fn"
server_key: "819448bbd27c526dc767d32013f577c6"
server_key_iv: "0c10501b5715d14773f24c693d9e2f0c"
server_key_version: "encryptionkey@0"
I20260925 07:34:11.223394 27647 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 07:34:11.224009 27662 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:34:11.224184 27647 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 07:34:11.224283 27647 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790321392278483-9480-0/minicluster-data/ts-0/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790321392278483-9480-0/minicluster-data/ts-0/wal
uuid: "0400dbd9ade1475b9078db59ef3ff5c2"
format_stamp: "Formatted at 2026-09-25 07:34:11 on dist-test-slave-57fn"
server_key: "819448bbd27c526dc767d32013f577c6"
server_key_iv: "0c10501b5715d14773f24c693d9e2f0c"
server_key_version: "encryptionkey@0"
I20260925 07:34:11.224362 27647 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790321392278483-9480-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790321392278483-9480-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790321392278483-9480-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:34:11.235526 27647 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:34:11.237401 27647 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:34:11.237607 27647 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:34:11.237874 27647 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:34:11.238175 27647 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:34:11.238227 27647 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:34:11.238258 27647 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:34:11.238283 27647 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:34:11.244141 27647 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.1:46773
I20260925 07:34:11.244181 27775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.1:46773 every 8 connection(s)
I20260925 07:34:11.244513 27647 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790321392278483-9480-0/minicluster-data/ts-0/data/info.pb
Sep 25 07:34:11 dist-test-slave-57fn krb5kdc[27540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321651, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.1@KRBTEST.COM for kudu/127.9.66.62@KRBTEST.COM
I20260925 07:34:11.252449 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 27647
I20260925 07:34:11.252537 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790321392278483-9480-0/minicluster-data/ts-0/wal/instance
I20260925 07:34:11.252774 9480 external_mini_cluster.cc:1428] Setting key abbe6291f8567847ed4df90a39df5dec
I20260925 07:34:11.253965 27625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:11.245229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.1:60279 (local address 127.9.66.62:42283)
0925 07:34:11.245309 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:11.245312 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:11.245584 (+ 272us) server_negotiation.cc:407] Connection header received
0925 07:34:11.245991 (+ 407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:11.245992 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:11.246005 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:11.246060 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:34:11.246973 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:11.247485 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:11.248404 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:11.248601 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:11.248653 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:11.250899 (+ 2246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:34:11.250900 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:34:11.250900 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:34:11.250907 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:34:11.252195 (+ 1288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:34:11.252700 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:34:11.252702 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:34:11.252703 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:34:11.252764 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:34:11.253105 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:34:11.253106 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:34:11.253107 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:34:11.253252 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:34:11.253306 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:11.253739 (+ 433us) server_negotiation.cc:299] Negotiation successful
0925 07:34:11.253835 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 07:34:11.253976 27778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:11.245169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:42283 (local address 127.9.66.1:60279)
0925 07:34:11.245391 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0925 07:34:11.245431 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 07:34:11.245912 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:34:11.246128 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:34:11.246131 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:34:11.246700 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:34:11.246882 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:11.246888 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:11.247563 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:11.247564 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:11.248326 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:11.248328 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:11.248672 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:11.248686 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:11.248779 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:11.249289 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:34:11.249296 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:34:11.250806 (+ 1510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:34:11.252305 (+ 1499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:34:11.252311 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:34:11.252313 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:34:11.252628 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:34:11.252849 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:34:11.252849 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:34:11.252850 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:34:11.252986 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:34:11.253321 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:34:11.253327 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:34:11.253586 (+ 259us) client_negotiation.cc:769] Sending connection context
0925 07:34:11.253701 (+ 115us) client_negotiation.cc:240] Negotiation successful
0925 07:34:11.253818 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":92,"threads_started":1}
I20260925 07:34:11.254621 27776 heartbeater.cc:347] Connected to a master server at 127.9.66.62:42283
I20260925 07:34:11.254696 27776 heartbeater.cc:464] Registering TS with master...
I20260925 07:34:11.254868 27776 heartbeater.cc:511] Master 127.9.66.62:42283 requested a full tablet report, sending...
I20260925 07:34:11.255482 27585 ts_manager.cc:194] Registered new tserver with Master: 0400dbd9ade1475b9078db59ef3ff5c2 (127.9.66.1:46773)
I20260925 07:34:11.256304 27585 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.66.1@KRBTEST.COM'} at 127.9.66.1:60279
WARNING: no policy specified for kudu/127.9.66.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.66.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.66.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.2.keytab.
Entry for principal kudu/127.9.66.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.2.keytab.
I20260925 07:34:11.280166 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790321392278483-9480-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790321392278483-9480-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790321392278483-9480-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790321392278483-9480-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.66.2:0
--local_ip_for_outbound_sockets=127.9.66.2
--webserver_interface=127.9.66.2
--webserver_port=0
--tserver_master_addrs=127.9.66.62:42283
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.2.keytab
--principal=kudu/127.9.66.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:36687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:34:11.359891 27781 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:34:11.360082 27781 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:34:11.360105 27781 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:34:11.361261 27781 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:34:11.361300 27781 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:34:11.361325 27781 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.66.2
I20260925 07:34:11.362780 27781 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:36687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790321392278483-9480-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790321392278483-9480-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.2.keytab
--principal=kudu/127.9.66.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.66.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790321392278483-9480-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.66.2
--webserver_port=0
--tserver_master_addrs=127.9.66.62:42283
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27781
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.66.2
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790321392278483-9480-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:34:11.363024 27781 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:34:11.363243 27781 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:34:11.368261 27789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:34:11.368261 27787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:34:11.368345 27786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:34:11.368518 27781 server_base.cc:1034] running on GCE node
I20260925 07:34:11.368707 27781 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:34:11.368891 27781 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:34:11.370038 27781 hybrid_clock.cc:648] HybridClock initialized: now 1790321651370019 us; error 35 us; skew 500 ppm
Sep 25 07:34:11 dist-test-slave-57fn krb5kdc[27540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321651, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:34:11.372572 27781 init.cc:377] Logged in from keytab as kudu/127.9.66.2@KRBTEST.COM (short username kudu)
I20260925 07:34:11.373034 27781 webserver.cc:460] Webserver started at http://127.9.66.2:46277/ using document root <none> and password file <none>
I20260925 07:34:11.373256 27781 fs_manager.cc:359] Metadata directory not provided
I20260925 07:34:11.373301 27781 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:34:11.373425 27781 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:34:11.374420 27781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790321392278483-9480-0/minicluster-data/ts-1/data/instance:
uuid: "02cb4a2d8d364077818041511d63daf2"
format_stamp: "Formatted at 2026-09-25 07:34:11 on dist-test-slave-57fn"
server_key: "f599107bad87e601ad1ef9462caa4802"
server_key_iv: "9120c6df4f7dad433bad626dd2e42e60"
server_key_version: "encryptionkey@0"
I20260925 07:34:11.374758 27781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790321392278483-9480-0/minicluster-data/ts-1/wal/instance:
uuid: "02cb4a2d8d364077818041511d63daf2"
format_stamp: "Formatted at 2026-09-25 07:34:11 on dist-test-slave-57fn"
server_key: "f599107bad87e601ad1ef9462caa4802"
server_key_iv: "9120c6df4f7dad433bad626dd2e42e60"
server_key_version: "encryptionkey@0"
I20260925 07:34:11.376056 27781 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 07:34:11.376818 27796 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:34:11.377002 27781 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 07:34:11.377099 27781 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790321392278483-9480-0/minicluster-data/ts-1/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790321392278483-9480-0/minicluster-data/ts-1/wal
uuid: "02cb4a2d8d364077818041511d63daf2"
format_stamp: "Formatted at 2026-09-25 07:34:11 on dist-test-slave-57fn"
server_key: "f599107bad87e601ad1ef9462caa4802"
server_key_iv: "9120c6df4f7dad433bad626dd2e42e60"
server_key_version: "encryptionkey@0"
I20260925 07:34:11.377174 27781 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790321392278483-9480-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790321392278483-9480-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790321392278483-9480-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:34:11.404078 27781 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:34:11.406522 27781 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:34:11.406687 27781 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:34:11.406934 27781 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:34:11.407253 27781 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:34:11.407306 27781 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:34:11.407346 27781 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:34:11.407377 27781 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:34:11.413332 27781 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.2:35025
I20260925 07:34:11.413384 27909 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.2:35025 every 8 connection(s)
I20260925 07:34:11.413715 27781 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790321392278483-9480-0/minicluster-data/ts-1/data/info.pb
I20260925 07:34:11.415053 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 27781
I20260925 07:34:11.415175 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790321392278483-9480-0/minicluster-data/ts-1/wal/instance
I20260925 07:34:11.415402 9480 external_mini_cluster.cc:1428] Setting key dfb33a5187adcc2b8734d36c06806228
Sep 25 07:34:11 dist-test-slave-57fn krb5kdc[27540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321651, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.2@KRBTEST.COM for kudu/127.9.66.62@KRBTEST.COM
I20260925 07:34:11.424100 27912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:11.414388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:42283 (local address 127.9.66.2:58015)
0925 07:34:11.414599 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0925 07:34:11.414652 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 07:34:11.415480 (+ 828us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:34:11.415813 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:34:11.415817 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:34:11.416694 (+ 877us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:34:11.416990 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:11.416996 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:11.417733 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:11.417735 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:11.418544 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:11.418546 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:11.418944 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:11.418945 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:11.419071 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:11.419541 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:34:11.419549 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:34:11.420911 (+ 1362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:34:11.422782 (+ 1871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:34:11.422787 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:34:11.422788 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:34:11.423003 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:34:11.423241 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:34:11.423241 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:34:11.423241 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:34:11.423331 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:34:11.423648 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:34:11.423652 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:34:11.423755 (+ 103us) client_negotiation.cc:769] Sending connection context
0925 07:34:11.423832 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 07:34:11.423951 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":95,"threads_started":1}
I20260925 07:34:11.424105 27625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:11.414473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.2:58015 (local address 127.9.66.62:42283)
0925 07:34:11.414568 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:11.414571 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:11.414740 (+ 169us) server_negotiation.cc:407] Connection header received
0925 07:34:11.415633 (+ 893us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:11.415635 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:11.415647 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:11.415702 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:34:11.417089 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:11.417640 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:11.418632 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:11.418833 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:11.418931 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:11.421006 (+ 2075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:34:11.421008 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:34:11.421008 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:34:11.421017 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:34:11.422697 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:34:11.423080 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:34:11.423081 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:34:11.423082 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:34:11.423144 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:34:11.423415 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:34:11.423416 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:34:11.423417 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:34:11.423571 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:34:11.423664 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:11.423896 (+ 232us) server_negotiation.cc:299] Negotiation successful
0925 07:34:11.423993 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
WARNING: no policy specified for kudu/127.9.66.3@KRBTEST.COM; defaulting to no policy
I20260925 07:34:11.424700 27910 heartbeater.cc:347] Connected to a master server at 127.9.66.62:42283
I20260925 07:34:11.424775 27910 heartbeater.cc:464] Registering TS with master...
I20260925 07:34:11.424970 27910 heartbeater.cc:511] Master 127.9.66.62:42283 requested a full tablet report, sending...
I20260925 07:34:11.425320 27585 ts_manager.cc:194] Registered new tserver with Master: 02cb4a2d8d364077818041511d63daf2 (127.9.66.2:35025)
I20260925 07:34:11.425798 27585 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.66.2@KRBTEST.COM'} at 127.9.66.2:58015
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.66.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.66.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.3.keytab.
Entry for principal kudu/127.9.66.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.3.keytab.
I20260925 07:34:11.444001 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790321392278483-9480-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790321392278483-9480-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790321392278483-9480-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790321392278483-9480-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.66.3:0
--local_ip_for_outbound_sockets=127.9.66.3
--webserver_interface=127.9.66.3
--webserver_port=0
--tserver_master_addrs=127.9.66.62:42283
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.3.keytab
--principal=kudu/127.9.66.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:36687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:34:11.524719 27915 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:34:11.524910 27915 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:34:11.524932 27915 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:34:11.526072 27915 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:34:11.526113 27915 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:34:11.526137 27915 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.66.3
I20260925 07:34:11.527684 27915 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:36687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790321392278483-9480-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790321392278483-9480-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.3.keytab
--principal=kudu/127.9.66.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.66.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790321392278483-9480-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.66.3
--webserver_port=0
--tserver_master_addrs=127.9.66.62:42283
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27915
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.66.3
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790321392278483-9480-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:34:11.527887 27915 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:34:11.528090 27915 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:34:11.532914 27920 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:34:11.533022 27915 server_base.cc:1034] running on GCE node
W20260925 07:34:11.532913 27921 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:34:11.532915 27923 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:34:11.533321 27915 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:34:11.533612 27915 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:34:11.534755 27915 hybrid_clock.cc:648] HybridClock initialized: now 1790321651534736 us; error 38 us; skew 500 ppm
Sep 25 07:34:11 dist-test-slave-57fn krb5kdc[27540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321651, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:34:11.537228 27915 init.cc:377] Logged in from keytab as kudu/127.9.66.3@KRBTEST.COM (short username kudu)
I20260925 07:34:11.537757 27915 webserver.cc:460] Webserver started at http://127.9.66.3:46493/ using document root <none> and password file <none>
I20260925 07:34:11.538017 27915 fs_manager.cc:359] Metadata directory not provided
I20260925 07:34:11.538079 27915 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:34:11.538208 27915 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:34:11.539243 27915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790321392278483-9480-0/minicluster-data/ts-2/data/instance:
uuid: "b918c2f55bee4ffeae7195a571e8f5c7"
format_stamp: "Formatted at 2026-09-25 07:34:11 on dist-test-slave-57fn"
server_key: "78ea2866a856f8bd0ecb825d084ba80a"
server_key_iv: "d9ad21a6d4563301cbba249bdde5174b"
server_key_version: "encryptionkey@0"
I20260925 07:34:11.539613 27915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790321392278483-9480-0/minicluster-data/ts-2/wal/instance:
uuid: "b918c2f55bee4ffeae7195a571e8f5c7"
format_stamp: "Formatted at 2026-09-25 07:34:11 on dist-test-slave-57fn"
server_key: "78ea2866a856f8bd0ecb825d084ba80a"
server_key_iv: "d9ad21a6d4563301cbba249bdde5174b"
server_key_version: "encryptionkey@0"
I20260925 07:34:11.540925 27915 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 07:34:11.541690 27930 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:34:11.541874 27915 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:34:11.541960 27915 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790321392278483-9480-0/minicluster-data/ts-2/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790321392278483-9480-0/minicluster-data/ts-2/wal
uuid: "b918c2f55bee4ffeae7195a571e8f5c7"
format_stamp: "Formatted at 2026-09-25 07:34:11 on dist-test-slave-57fn"
server_key: "78ea2866a856f8bd0ecb825d084ba80a"
server_key_iv: "d9ad21a6d4563301cbba249bdde5174b"
server_key_version: "encryptionkey@0"
I20260925 07:34:11.542043 27915 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790321392278483-9480-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790321392278483-9480-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790321392278483-9480-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:34:11.570212 27915 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:34:11.572034 27915 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:34:11.572194 27915 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:34:11.572420 27915 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:34:11.572745 27915 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:34:11.572793 27915 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:34:11.572832 27915 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:34:11.572876 27915 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:34:11.578490 27915 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.3:43425
I20260925 07:34:11.578557 28043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.3:43425 every 8 connection(s)
I20260925 07:34:11.578860 27915 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790321392278483-9480-0/minicluster-data/ts-2/data/info.pb
Sep 25 07:34:11 dist-test-slave-57fn krb5kdc[27540](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321651, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.3@KRBTEST.COM for kudu/127.9.66.62@KRBTEST.COM
I20260925 07:34:11.587903 27625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:11.579555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.3:59863 (local address 127.9.66.62:42283)
0925 07:34:11.579598 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:11.579601 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:11.579814 (+ 213us) server_negotiation.cc:407] Connection header received
0925 07:34:11.580305 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:11.580306 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:11.580316 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:11.580366 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:34:11.581342 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:11.581876 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:11.582922 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:11.583104 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:11.583153 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:11.585204 (+ 2051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:34:11.585206 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:34:11.585206 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:34:11.585212 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:34:11.586631 (+ 1419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:34:11.587024 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:34:11.587025 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:34:11.587025 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:34:11.587075 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:34:11.587295 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:34:11.587295 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:34:11.587295 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:34:11.587394 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:34:11.587447 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:11.587754 (+ 307us) server_negotiation.cc:299] Negotiation successful
0925 07:34:11.587821 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 07:34:11.588020 28046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:11.579537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:42283 (local address 127.9.66.3:59863)
0925 07:34:11.579675 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 07:34:11.579708 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 07:34:11.580224 (+ 516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:34:11.580457 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:34:11.580460 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:34:11.581086 (+ 626us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:34:11.581262 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:11.581266 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:11.581974 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:11.581975 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:11.582846 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:11.582848 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:11.583171 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:11.583172 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:11.583267 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:11.583785 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:34:11.583792 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:34:11.585116 (+ 1324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:34:11.586713 (+ 1597us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:34:11.586718 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:34:11.586720 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:34:11.586952 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:34:11.587146 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:34:11.587147 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:34:11.587147 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:34:11.587241 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:34:11.587470 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:34:11.587475 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:34:11.587599 (+ 124us) client_negotiation.cc:769] Sending connection context
0925 07:34:11.587694 (+ 95us) client_negotiation.cc:240] Negotiation successful
0925 07:34:11.587814 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":66,"threads_started":1}
I20260925 07:34:11.588480 28044 heartbeater.cc:347] Connected to a master server at 127.9.66.62:42283
I20260925 07:34:11.588591 28044 heartbeater.cc:464] Registering TS with master...
I20260925 07:34:11.588766 28044 heartbeater.cc:511] Master 127.9.66.62:42283 requested a full tablet report, sending...
I20260925 07:34:11.588768 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 27915
I20260925 07:34:11.588855 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790321392278483-9480-0/minicluster-data/ts-2/wal/instance
I20260925 07:34:11.589048 9480 external_mini_cluster.cc:1428] Setting key 52c0024c827cd29724e1a87722618220
I20260925 07:34:11.589190 27585 ts_manager.cc:194] Registered new tserver with Master: b918c2f55bee4ffeae7195a571e8f5c7 (127.9.66.3:43425)
I20260925 07:34:11.589676 27585 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.66.3@KRBTEST.COM'} at 127.9.66.3:59863
I20260925 07:34:11.589983 9480 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 07:34:11.597152 27625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:11.591260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39844 (local address 127.9.66.62:42283)
0925 07:34:11.591323 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:11.591324 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:11.591448 (+ 124us) server_negotiation.cc:407] Connection header received
0925 07:34:11.591519 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:11.591519 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:11.591527 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:11.591569 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:34:11.592156 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:11.592607 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:11.593411 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:11.593646 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:11.593698 (+ 52us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:11.595060 (+ 1362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:34:11.595061 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:34:11.595061 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:34:11.595067 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:34:11.596161 (+ 1094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:34:11.596538 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:34:11.596539 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:34:11.596539 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:34:11.596604 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:34:11.596789 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:34:11.596789 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:34:11.596790 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:34:11.596905 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:34:11.596978 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:11.597057 (+ 79us) server_negotiation.cc:299] Negotiation successful
0925 07:34:11.597099 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 07:34:11.598645 27584 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39844:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 07:34:11.600119 27584 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 07:34:11.606472 28055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:11.602295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.1:46773 (local address 127.0.0.1:56808)
0925 07:34:11.602641 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0925 07:34:11.602681 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 07:34:11.602749 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:34:11.602945 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:34:11.602948 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:34:11.602950 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:34:11.603016 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:11.603018 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:11.604086 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:11.604087 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:11.604994 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:11.604996 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:11.606133 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:11.606135 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:11.606229 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:11.606245 (+ 16us) client_negotiation.cc:769] Sending connection context
0925 07:34:11.606329 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 07:34:11.606394 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"mutex_wait_us":3,"thread_start_us":52,"threads_started":1}
I20260925 07:34:11.606487 28057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:11.602453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56808 (local address 127.9.66.1:46773)
0925 07:34:11.602785 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:11.602789 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:11.602799 (+ 10us) server_negotiation.cc:407] Connection header received
0925 07:34:11.602837 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:11.602838 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:11.602851 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:11.602943 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:34:11.603120 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:11.603991 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:11.605133 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:11.605954 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:11.606037 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:11.606093 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:11.606349 (+ 256us) server_negotiation.cc:299] Negotiation successful
0925 07:34:11.606396 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":88,"threads_started":1}
I20260925 07:34:11.606887 28056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:11.602411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.2:35025 (local address 127.0.0.1:44802)
0925 07:34:11.602648 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0925 07:34:11.602685 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 07:34:11.602750 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:34:11.603083 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:34:11.603084 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:34:11.603086 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:34:11.603143 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:11.603145 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:11.604735 (+ 1590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:11.604736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:11.605722 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:11.605724 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:11.606506 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:11.606507 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:11.606731 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:11.606735 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 07:34:11.606781 (+ 46us) client_negotiation.cc:240] Negotiation successful
0925 07:34:11.606820 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"mutex_wait_us":37,"thread_start_us":104,"threads_started":1}
I20260925 07:34:11.607126 28060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:11.602688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44802 (local address 127.9.66.2:35025)
0925 07:34:11.602913 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:11.602916 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:11.602925 (+ 9us) server_negotiation.cc:407] Connection header received
0925 07:34:11.602962 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:11.602964 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:11.602976 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:11.603085 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:34:11.603405 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:11.604616 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:11.605816 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:11.606348 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:11.606420 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:11.606473 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:11.606986 (+ 513us) server_negotiation.cc:299] Negotiation successful
0925 07:34:11.607031 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":94,"threads_started":1}
I20260925 07:34:11.607319 27710 tablet_service.cc:1467] Processing CreateTablet for tablet f3af9c3f1b094fa48f9f87a0d4b2bd5b (DEFAULT_TABLE table=test-table [id=a13b1b4bd2e243dc9ad5a6b93eb7746e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:34:11.607671 28058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:11.602498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.3:43425 (local address 127.0.0.1:53254)
0925 07:34:11.603202 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0925 07:34:11.603213 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 07:34:11.603261 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:34:11.603478 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:34:11.603478 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:34:11.603481 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:34:11.603532 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:11.603533 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:11.604913 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:11.604914 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:11.606162 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:11.606164 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:11.607461 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:11.607462 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:11.607521 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:11.607525 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 07:34:11.607573 (+ 48us) client_negotiation.cc:240] Negotiation successful
0925 07:34:11.607610 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":685,"thread_start_us":83,"threads_started":1}
I20260925 07:34:11.607758 27710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3af9c3f1b094fa48f9f87a0d4b2bd5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:34:11.607888 28059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:11.602560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53254 (local address 127.9.66.3:43425)
0925 07:34:11.602699 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:11.602704 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:11.603293 (+ 589us) server_negotiation.cc:407] Connection header received
0925 07:34:11.603328 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:11.603330 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:11.603342 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:11.603420 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:34:11.603622 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:11.604754 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:11.606258 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:11.607243 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:11.607335 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:11.607387 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:11.607767 (+ 380us) server_negotiation.cc:299] Negotiation successful
0925 07:34:11.607809 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":60,"threads_started":1}
I20260925 07:34:11.607831 27844 tablet_service.cc:1467] Processing CreateTablet for tablet f3af9c3f1b094fa48f9f87a0d4b2bd5b (DEFAULT_TABLE table=test-table [id=a13b1b4bd2e243dc9ad5a6b93eb7746e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:34:11.608130 27844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3af9c3f1b094fa48f9f87a0d4b2bd5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:34:11.610378 27978 tablet_service.cc:1467] Processing CreateTablet for tablet f3af9c3f1b094fa48f9f87a0d4b2bd5b (DEFAULT_TABLE table=test-table [id=a13b1b4bd2e243dc9ad5a6b93eb7746e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:34:11.610685 27978 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3af9c3f1b094fa48f9f87a0d4b2bd5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:34:11.610870 28061 tablet_bootstrap.cc:492] T f3af9c3f1b094fa48f9f87a0d4b2bd5b P 02cb4a2d8d364077818041511d63daf2: Bootstrap starting.
I20260925 07:34:11.611270 28062 tablet_bootstrap.cc:492] T f3af9c3f1b094fa48f9f87a0d4b2bd5b P 0400dbd9ade1475b9078db59ef3ff5c2: Bootstrap starting.
I20260925 07:34:11.611691 28061 tablet_bootstrap.cc:654] T f3af9c3f1b094fa48f9f87a0d4b2bd5b P 02cb4a2d8d364077818041511d63daf2: Neither blocks nor log segments found. Creating new log.
I20260925 07:34:11.612135 28062 tablet_bootstrap.cc:654] T f3af9c3f1b094fa48f9f87a0d4b2bd5b P 0400dbd9ade1475b9078db59ef3ff5c2: Neither blocks nor log segments found. Creating new log.
I20260925 07:34:11.612396 28061 log.cc:824] T f3af9c3f1b094fa48f9f87a0d4b2bd5b P 02cb4a2d8d364077818041511d63daf2: Log is configured to *not* fsync() on all Append() calls
I20260925 07:34:11.612941 28062 log.cc:824] T f3af9c3f1b094fa48f9f87a0d4b2bd5b P 0400dbd9ade1475b9078db59ef3ff5c2: Log is configured to *not* fsync() on all Append() calls
I20260925 07:34:11.613433 28061 tablet_bootstrap.cc:492] T f3af9c3f1b094fa48f9f87a0d4b2bd5b P 02cb4a2d8d364077818041511d63daf2: No bootstrap required, opened a new log
I20260925 07:34:11.613549 28061 ts_tablet_manager.cc:1397] T f3af9c3f1b094fa48f9f87a0d4b2bd5b P 02cb4a2d8d364077818041511d63daf2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 07:34:11.613832 28062 tablet_bootstrap.cc:492] T f3af9c3f1b094fa48f9f87a0d4b2bd5b P 0400dbd9ade1475b9078db59ef3ff5c2: No bootstrap required, opened a new log
I20260925 07:34:11.613914 28062 ts_tablet_manager.cc:1397] T f3af9c3f1b094fa48f9f87a0d4b2bd5b P 0400dbd9ade1475b9078db59ef3ff5c2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 07:34:11.613957 28065 tablet_bootstrap.cc:492] T f3af9c3f1b094fa48f9f87a0d4b2bd5b P b918c2f55bee4ffeae7195a571e8f5c7: Bootstrap starting.
I20260925 07:34:11.614444 28065 tablet_bootstrap.cc:654] T f3af9c3f1b094fa48f9f87a0d4b2bd5b P b918c2f55bee4ffeae7195a571e8f5c7: Neither blocks nor log segments found. Creating new log.
I20260925 07:34:11.614936 28065 log.cc:824] T f3af9c3f1b094fa48f9f87a0d4b2bd5b P b918c2f55bee4ffeae7195a571e8f5c7: Log is configured to *not* fsync() on all Append() calls
I20260925 07:34:11.615564 28065 tablet_bootstrap.cc:492] T f3af9c3f1b094fa48f9f87a0d4b2bd5b P b918c2f55bee4ffeae7195a571e8f5c7: No bootstrap required, opened a new log
I20260925 07:34:11.615639 28065 ts_tablet_manager.cc:1397] T f3af9c3f1b094fa48f9f87a0d4b2bd5b P b918c2f55bee4ffeae7195a571e8f5c7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 07:34:11.616214 28062 raft_consensus.cc:348] T f3af9c3f1b094fa48f9f87a0d4b2bd5b P 0400dbd9ade1475b9078db59ef3ff5c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0400dbd9ade1475b9078db59ef3ff5c2" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 46773 } } peers { permanent_uuid: "b918c2f55bee4ffeae7195a571e8f5c7" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 43425 } } peers { permanent_uuid: "02cb4a2d8d364077818041511d63daf2" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 35025 } }
I20260925 07:34:11.616442 28062 raft_consensus.cc:374] T f3af9c3f1b094fa48f9f87a0d4b2bd5b P 0400dbd9ade1475b9078db59ef3ff5c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:34:11.616503 28062 raft_consensus.cc:729] T f3af9c3f1b094fa48f9f87a0d4b2bd5b P 0400dbd9ade1475b9078db59ef3ff5c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0400dbd9ade1475b9078db59ef3ff5c2, State: Initialized, Role: FOLLOWER
I20260925 07:34:11.616050 28061 raft_consensus.cc:348] T f3af9c3f1b094fa48f9f87a0d4b2bd5b P 02cb4a2d8d364077818041511d63daf2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0400dbd9ade1475b9078db59ef3ff5c2" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 46773 } } peers { permanent_uuid: "b918c2f55bee4ffeae7195a571e8f5c7" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 43425 } } peers { permanent_uuid: "02cb4a2d8d364077818041511d63daf2" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 35025 } }
I20260925 07:34:11.616659 28061 raft_consensus.cc:374] T f3af9c3f1b094fa48f9f87a0d4b2bd5b P 02cb4a2d8d364077818041511d63daf2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:34:11.616658 28062 consensus_queue.cc:260] T f3af9c3f1b094fa48f9f87a0d4b2bd5b P 0400dbd9ade1475b9078db59ef3ff5c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0400dbd9ade1475b9078db59ef3ff5c2" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 46773 } } peers { permanent_uuid: "b918c2f55bee4ffeae7195a571e8f5c7" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 43425 } } peers { permanent_uuid: "02cb4a2d8d364077818041511d63daf2" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 35025 } }
I20260925 07:34:11.616708 28061 raft_consensus.cc:729] T f3af9c3f1b094fa48f9f87a0d4b2bd5b P 02cb4a2d8d364077818041511d63daf2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02cb4a2d8d364077818041511d63daf2, State: Initialized, Role: FOLLOWER
I20260925 07:34:11.616878 28061 consensus_queue.cc:260] T f3af9c3f1b094fa48f9f87a0d4b2bd5b P 02cb4a2d8d364077818041511d63daf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0400dbd9ade1475b9078db59ef3ff5c2" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 46773 } } peers { permanent_uuid: "b918c2f55bee4ffeae7195a571e8f5c7" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 43425 } } peers { permanent_uuid: "02cb4a2d8d364077818041511d63daf2" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 35025 } }
I20260925 07:34:11.616988 27776 heartbeater.cc:503] Master 127.9.66.62:42283 was elected leader, sending a full tablet report...
I20260925 07:34:11.617169 28061 ts_tablet_manager.cc:1428] T f3af9c3f1b094fa48f9f87a0d4b2bd5b P 02cb4a2d8d364077818041511d63daf2: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260925 07:34:11.617298 27910 heartbeater.cc:503] Master 127.9.66.62:42283 was elected leader, sending a full tablet report...
I20260925 07:34:11.616992 28062 ts_tablet_manager.cc:1428] T f3af9c3f1b094fa48f9f87a0d4b2bd5b P 0400dbd9ade1475b9078db59ef3ff5c2: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 07:34:11.617587 28065 raft_consensus.cc:348] T f3af9c3f1b094fa48f9f87a0d4b2bd5b P b918c2f55bee4ffeae7195a571e8f5c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0400dbd9ade1475b9078db59ef3ff5c2" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 46773 } } peers { permanent_uuid: "b918c2f55bee4ffeae7195a571e8f5c7" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 43425 } } peers { permanent_uuid: "02cb4a2d8d364077818041511d63daf2" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 35025 } }
I20260925 07:34:11.617758 28065 raft_consensus.cc:374] T f3af9c3f1b094fa48f9f87a0d4b2bd5b P b918c2f55bee4ffeae7195a571e8f5c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:34:11.617811 28065 raft_consensus.cc:729] T f3af9c3f1b094fa48f9f87a0d4b2bd5b P b918c2f55bee4ffeae7195a571e8f5c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b918c2f55bee4ffeae7195a571e8f5c7, State: Initialized, Role: FOLLOWER
I20260925 07:34:11.617928 28065 consensus_queue.cc:260] T f3af9c3f1b094fa48f9f87a0d4b2bd5b P b918c2f55bee4ffeae7195a571e8f5c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0400dbd9ade1475b9078db59ef3ff5c2" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 46773 } } peers { permanent_uuid: "b918c2f55bee4ffeae7195a571e8f5c7" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 43425 } } peers { permanent_uuid: "02cb4a2d8d364077818041511d63daf2" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 35025 } }
I20260925 07:34:11.618191 28044 heartbeater.cc:503] Master 127.9.66.62:42283 was elected leader, sending a full tablet report...
I20260925 07:34:11.618381 28065 ts_tablet_manager.cc:1428] T f3af9c3f1b094fa48f9f87a0d4b2bd5b P b918c2f55bee4ffeae7195a571e8f5c7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
W20260925 07:34:11.664263 27911 tablet.cc:2369] T f3af9c3f1b094fa48f9f87a0d4b2bd5b P 02cb4a2d8d364077818041511d63daf2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 07:34:11.745173 27777 tablet.cc:2369] T f3af9c3f1b094fa48f9f87a0d4b2bd5b P 0400dbd9ade1475b9078db59ef3ff5c2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 07:34:11.829414 28045 tablet.cc:2369] T f3af9c3f1b094fa48f9f87a0d4b2bd5b P b918c2f55bee4ffeae7195a571e8f5c7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 07:34:11.863577 28067 raft_consensus.cc:482] T f3af9c3f1b094fa48f9f87a0d4b2bd5b P 0400dbd9ade1475b9078db59ef3ff5c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 07:34:11.863715 28067 raft_consensus.cc:504] T f3af9c3f1b094fa48f9f87a0d4b2bd5b P 0400dbd9ade1475b9078db59ef3ff5c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0400dbd9ade1475b9078db59ef3ff5c2" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 46773 } } peers { permanent_uuid: "b918c2f55bee4ffeae7195a571e8f5c7" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 43425 } } peers { permanent_uuid: "02cb4a2d8d364077818041511d63daf2" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 35025 } }
I20260925 07:34:11.864127 28067 leader_election.cc:288] T f3af9c3f1b094fa48f9f87a0d4b2bd5b P 0400dbd9ade1475b9078db59ef3ff5c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b918c2f55bee4ffeae7195a571e8f5c7 (127.9.66.3:43425), 02cb4a2d8d364077818041511d63daf2 (127.9.66.2:35025)
I20260925 07:34:11.867023 28060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:11.864380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.1:42255 (local address 127.9.66.2:35025)
0925 07:34:11.864426 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:11.864428 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:11.864508 (+ 80us) server_negotiation.cc:407] Connection header received
0925 07:34:11.864586 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:11.864587 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:11.864599 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:11.864670 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:34:11.864830 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:11.865304 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:11.866256 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:11.866668 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:11.866723 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:11.866730 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:11.866909 (+ 179us) server_negotiation.cc:299] Negotiation successful
0925 07:34:11.866954 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 07:34:11.867079 28059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:11.864368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.1:42783 (local address 127.9.66.3:43425)
0925 07:34:11.864413 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:11.864415 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:11.864557 (+ 142us) server_negotiation.cc:407] Connection header received
0925 07:34:11.864592 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:11.864597 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:11.864606 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:11.864671 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:34:11.864858 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:11.865312 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:11.866315 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:11.866692 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:11.866784 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:11.866793 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:11.866977 (+ 184us) server_negotiation.cc:299] Negotiation successful
0925 07:34:11.867019 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 07:34:11.867074 28070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:11.864299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.2:35025 (local address 127.9.66.1:42255)
0925 07:34:11.864429 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 07:34:11.864438 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 07:34:11.864505 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:34:11.864668 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:34:11.864669 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:34:11.864671 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:34:11.864747 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:11.864749 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:11.865386 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:11.865387 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:11.866164 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:11.866166 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:11.866754 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:11.866755 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:11.866826 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:11.866830 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 07:34:11.866888 (+ 58us) client_negotiation.cc:240] Negotiation successful
0925 07:34:11.866917 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":86,"threads_started":1}
I20260925 07:34:11.867111 28071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:11.864299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.3:43425 (local address 127.9.66.1:42783)
0925 07:34:11.864443 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 07:34:11.864453 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 07:34:11.864545 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:34:11.864690 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:34:11.864690 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:34:11.864692 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:34:11.864767 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:11.864768 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:11.865434 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:11.865435 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:11.866224 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:11.866227 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:11.866808 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:11.866809 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:11.866888 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:11.866893 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 07:34:11.866950 (+ 57us) client_negotiation.cc:240] Negotiation successful
0925 07:34:11.866986 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"mutex_wait_us":2,"thread_start_us":67,"threads_started":1}
I20260925 07:34:11.867455 27864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3af9c3f1b094fa48f9f87a0d4b2bd5b" candidate_uuid: "0400dbd9ade1475b9078db59ef3ff5c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02cb4a2d8d364077818041511d63daf2" is_pre_election: true
I20260925 07:34:11.867625 27864 raft_consensus.cc:2454] T f3af9c3f1b094fa48f9f87a0d4b2bd5b P 02cb4a2d8d364077818041511d63daf2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0400dbd9ade1475b9078db59ef3ff5c2 in term 0.
I20260925 07:34:11.867730 27998 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3af9c3f1b094fa48f9f87a0d4b2bd5b" candidate_uuid: "0400dbd9ade1475b9078db59ef3ff5c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b918c2f55bee4ffeae7195a571e8f5c7" is_pre_election: true
I20260925 07:34:11.867858 27665 leader_election.cc:302] T f3af9c3f1b094fa48f9f87a0d4b2bd5b P 0400dbd9ade1475b9078db59ef3ff5c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02cb4a2d8d364077818041511d63daf2, 0400dbd9ade1475b9078db59ef3ff5c2; no voters:
I20260925 07:34:11.867887 27998 raft_consensus.cc:2454] T f3af9c3f1b094fa48f9f87a0d4b2bd5b P b918c2f55bee4ffeae7195a571e8f5c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0400dbd9ade1475b9078db59ef3ff5c2 in term 0.
I20260925 07:34:11.868003 28067 raft_consensus.cc:2781] T f3af9c3f1b094fa48f9f87a0d4b2bd5b P 0400dbd9ade1475b9078db59ef3ff5c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 07:34:11.868067 28067 raft_consensus.cc:482] T f3af9c3f1b094fa48f9f87a0d4b2bd5b P 0400dbd9ade1475b9078db59ef3ff5c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 07:34:11.868090 28067 raft_consensus.cc:3037] T f3af9c3f1b094fa48f9f87a0d4b2bd5b P 0400dbd9ade1475b9078db59ef3ff5c2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:34:11.868772 28067 raft_consensus.cc:504] T f3af9c3f1b094fa48f9f87a0d4b2bd5b P 0400dbd9ade1475b9078db59ef3ff5c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0400dbd9ade1475b9078db59ef3ff5c2" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 46773 } } peers { permanent_uuid: "b918c2f55bee4ffeae7195a571e8f5c7" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 43425 } } peers { permanent_uuid: "02cb4a2d8d364077818041511d63daf2" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 35025 } }
I20260925 07:34:11.868907 28067 leader_election.cc:288] T f3af9c3f1b094fa48f9f87a0d4b2bd5b P 0400dbd9ade1475b9078db59ef3ff5c2 [CANDIDATE]: Term 1 election: Requested vote from peers b918c2f55bee4ffeae7195a571e8f5c7 (127.9.66.3:43425), 02cb4a2d8d364077818041511d63daf2 (127.9.66.2:35025)
I20260925 07:34:11.869076 27864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3af9c3f1b094fa48f9f87a0d4b2bd5b" candidate_uuid: "0400dbd9ade1475b9078db59ef3ff5c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02cb4a2d8d364077818041511d63daf2"
I20260925 07:34:11.869097 27998 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3af9c3f1b094fa48f9f87a0d4b2bd5b" candidate_uuid: "0400dbd9ade1475b9078db59ef3ff5c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b918c2f55bee4ffeae7195a571e8f5c7"
I20260925 07:34:11.869172 27864 raft_consensus.cc:3037] T f3af9c3f1b094fa48f9f87a0d4b2bd5b P 02cb4a2d8d364077818041511d63daf2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:34:11.869168 27998 raft_consensus.cc:3037] T f3af9c3f1b094fa48f9f87a0d4b2bd5b P b918c2f55bee4ffeae7195a571e8f5c7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:34:11.869982 27998 raft_consensus.cc:2454] T f3af9c3f1b094fa48f9f87a0d4b2bd5b P b918c2f55bee4ffeae7195a571e8f5c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0400dbd9ade1475b9078db59ef3ff5c2 in term 1.
I20260925 07:34:11.869993 27864 raft_consensus.cc:2454] T f3af9c3f1b094fa48f9f87a0d4b2bd5b P 02cb4a2d8d364077818041511d63daf2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0400dbd9ade1475b9078db59ef3ff5c2 in term 1.
I20260925 07:34:11.870151 27666 leader_election.cc:302] T f3af9c3f1b094fa48f9f87a0d4b2bd5b P 0400dbd9ade1475b9078db59ef3ff5c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0400dbd9ade1475b9078db59ef3ff5c2, b918c2f55bee4ffeae7195a571e8f5c7; no voters:
I20260925 07:34:11.870306 28067 raft_consensus.cc:2781] T f3af9c3f1b094fa48f9f87a0d4b2bd5b P 0400dbd9ade1475b9078db59ef3ff5c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:34:11.870496 28067 raft_consensus.cc:686] T f3af9c3f1b094fa48f9f87a0d4b2bd5b P 0400dbd9ade1475b9078db59ef3ff5c2 [term 1 LEADER]: Becoming Leader. State: Replica: 0400dbd9ade1475b9078db59ef3ff5c2, State: Running, Role: LEADER
I20260925 07:34:11.870596 28067 consensus_queue.cc:237] T f3af9c3f1b094fa48f9f87a0d4b2bd5b P 0400dbd9ade1475b9078db59ef3ff5c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0400dbd9ade1475b9078db59ef3ff5c2" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 46773 } } peers { permanent_uuid: "b918c2f55bee4ffeae7195a571e8f5c7" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 43425 } } peers { permanent_uuid: "02cb4a2d8d364077818041511d63daf2" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 35025 } }
I20260925 07:34:11.871438 27585 catalog_manager.cc:5495] T f3af9c3f1b094fa48f9f87a0d4b2bd5b P 0400dbd9ade1475b9078db59ef3ff5c2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0400dbd9ade1475b9078db59ef3ff5c2 (127.9.66.1). New cstate: current_term: 1 leader_uuid: "0400dbd9ade1475b9078db59ef3ff5c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0400dbd9ade1475b9078db59ef3ff5c2" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 46773 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b918c2f55bee4ffeae7195a571e8f5c7" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 43425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02cb4a2d8d364077818041511d63daf2" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 35025 } health_report { overall_health: UNKNOWN } } }
I20260925 07:34:11.890169 28057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:11.887929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56822 (local address 127.9.66.1:46773)
0925 07:34:11.887987 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:11.887989 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:11.887996 (+ 7us) server_negotiation.cc:407] Connection header received
0925 07:34:11.888030 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:11.888031 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:11.888049 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:11.888106 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 07:34:11.888213 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:11.888691 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:11.889417 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:11.889626 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:11.889717 (+ 91us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:11.889813 (+ 96us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 07:34:11.889893 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 07:34:11.890023 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:11.890051 (+ 28us) server_negotiation.cc:299] Negotiation successful
0925 07:34:11.890075 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 07:34:11.892022 27864 raft_consensus.cc:1261] T f3af9c3f1b094fa48f9f87a0d4b2bd5b P 02cb4a2d8d364077818041511d63daf2 [term 1 FOLLOWER]: Refusing update from remote peer 0400dbd9ade1475b9078db59ef3ff5c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:34:11.892031 27998 raft_consensus.cc:1261] T f3af9c3f1b094fa48f9f87a0d4b2bd5b P b918c2f55bee4ffeae7195a571e8f5c7 [term 1 FOLLOWER]: Refusing update from remote peer 0400dbd9ade1475b9078db59ef3ff5c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:34:11.892372 28072 consensus_queue.cc:1035] T f3af9c3f1b094fa48f9f87a0d4b2bd5b P 0400dbd9ade1475b9078db59ef3ff5c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b918c2f55bee4ffeae7195a571e8f5c7" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 43425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:34:11.892501 28067 consensus_queue.cc:1035] T f3af9c3f1b094fa48f9f87a0d4b2bd5b P 0400dbd9ade1475b9078db59ef3ff5c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02cb4a2d8d364077818041511d63daf2" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 35025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:34:11.895539 28076 mvcc.cc:205] Tried to move back new op lower bound from 7333157486146686976 to 7333157486062411776. Current Snapshot: MvccSnapshot[applied={T|T < 7333157486146686976 or (T in {7333157486146686976})}]
I20260925 07:34:11.897331 27584 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39844:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 07:34:11.897465 27584 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39844:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 07:34:11.898283 27584 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6bcfcc1d2cf640efbfbab1bc2633a576: Sending DeleteTablet for 3 replicas of tablet f3af9c3f1b094fa48f9f87a0d4b2bd5b
I20260925 07:34:11.898649 27844 tablet_service.cc:1514] Processing DeleteTablet for tablet f3af9c3f1b094fa48f9f87a0d4b2bd5b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:34:11 UTC) from {username='kudu', principal='kudu/127.9.66.62@KRBTEST.COM'} at 127.0.0.1:44802
I20260925 07:34:11.898705 27978 tablet_service.cc:1514] Processing DeleteTablet for tablet f3af9c3f1b094fa48f9f87a0d4b2bd5b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:34:11 UTC) from {username='kudu', principal='kudu/127.9.66.62@KRBTEST.COM'} at 127.0.0.1:53254
I20260925 07:34:11.898941 28085 tablet_replica.cc:330] T f3af9c3f1b094fa48f9f87a0d4b2bd5b P 02cb4a2d8d364077818041511d63daf2: stopping tablet replica
I20260925 07:34:11.898948 28086 tablet_replica.cc:330] T f3af9c3f1b094fa48f9f87a0d4b2bd5b P b918c2f55bee4ffeae7195a571e8f5c7: stopping tablet replica
I20260925 07:34:11.899041 28086 raft_consensus.cc:2229] T f3af9c3f1b094fa48f9f87a0d4b2bd5b P b918c2f55bee4ffeae7195a571e8f5c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 07:34:11.899021 27710 tablet_service.cc:1514] Processing DeleteTablet for tablet f3af9c3f1b094fa48f9f87a0d4b2bd5b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:34:11 UTC) from {username='kudu', principal='kudu/127.9.66.62@KRBTEST.COM'} at 127.0.0.1:56808
I20260925 07:34:11.899085 28085 raft_consensus.cc:2229] T f3af9c3f1b094fa48f9f87a0d4b2bd5b P 02cb4a2d8d364077818041511d63daf2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 07:34:11.899202 28085 raft_consensus.cc:2258] T f3af9c3f1b094fa48f9f87a0d4b2bd5b P 02cb4a2d8d364077818041511d63daf2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:34:11.899243 28086 raft_consensus.cc:2258] T f3af9c3f1b094fa48f9f87a0d4b2bd5b P b918c2f55bee4ffeae7195a571e8f5c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:34:11.899245 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 27647
I20260925 07:34:11.900076 28085 ts_tablet_manager.cc:1905] T f3af9c3f1b094fa48f9f87a0d4b2bd5b P 02cb4a2d8d364077818041511d63daf2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 07:34:11.900681 28086 ts_tablet_manager.cc:1905] T f3af9c3f1b094fa48f9f87a0d4b2bd5b P b918c2f55bee4ffeae7195a571e8f5c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 07:34:11.902112 28085 ts_tablet_manager.cc:1918] T f3af9c3f1b094fa48f9f87a0d4b2bd5b P 02cb4a2d8d364077818041511d63daf2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 07:34:11.902191 28085 log.cc:1196] T f3af9c3f1b094fa48f9f87a0d4b2bd5b P 02cb4a2d8d364077818041511d63daf2: Deleting WAL directory at /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790321392278483-9480-0/minicluster-data/ts-1/wal/wals/f3af9c3f1b094fa48f9f87a0d4b2bd5b
I20260925 07:34:11.902433 28085 ts_tablet_manager.cc:1939] T f3af9c3f1b094fa48f9f87a0d4b2bd5b P 02cb4a2d8d364077818041511d63daf2: Deleting consensus metadata
I20260925 07:34:11.902657 28086 ts_tablet_manager.cc:1918] T f3af9c3f1b094fa48f9f87a0d4b2bd5b P b918c2f55bee4ffeae7195a571e8f5c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 07:34:11.902752 28086 log.cc:1196] T f3af9c3f1b094fa48f9f87a0d4b2bd5b P b918c2f55bee4ffeae7195a571e8f5c7: Deleting WAL directory at /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790321392278483-9480-0/minicluster-data/ts-2/wal/wals/f3af9c3f1b094fa48f9f87a0d4b2bd5b
I20260925 07:34:11.902863 27571 catalog_manager.cc:4841] TS 02cb4a2d8d364077818041511d63daf2 (127.9.66.2:35025): tablet f3af9c3f1b094fa48f9f87a0d4b2bd5b (table test-table [id=a13b1b4bd2e243dc9ad5a6b93eb7746e]) successfully deleted
I20260925 07:34:11.903103 28086 ts_tablet_manager.cc:1939] T f3af9c3f1b094fa48f9f87a0d4b2bd5b P b918c2f55bee4ffeae7195a571e8f5c7: Deleting consensus metadata
I20260925 07:34:11.903528 27572 catalog_manager.cc:4841] TS b918c2f55bee4ffeae7195a571e8f5c7 (127.9.66.3:43425): tablet f3af9c3f1b094fa48f9f87a0d4b2bd5b (table test-table [id=a13b1b4bd2e243dc9ad5a6b93eb7746e]) successfully deleted
W20260925 07:34:11.905218 27572 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.66.1:46773 (error 108)
I20260925 07:34:11.905782 28058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:11.905636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.1:46773 (local address 127.0.0.1:56832)
0925 07:34:11.905705 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0925 07:34:11.905748 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.1:46773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260925 07:34:11.905777 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 27781
W20260925 07:34:11.905908 27572 catalog_manager.cc:4584] TS 0400dbd9ade1475b9078db59ef3ff5c2 (127.9.66.1:46773): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f3af9c3f1b094fa48f9f87a0d4b2bd5b: Network error: Client connection negotiation failed: client connection to 127.9.66.1:46773: connect: Connection refused (error 111)
I20260925 07:34:11.911509 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 27915
I20260925 07:34:11.915997 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 27553
2026-09-25T07:34:11Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2204 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-25T07:34:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T07:34:11Z Disabled control of system clock
I20260925 07:34:11.942982 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790321392278483-9480-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790321392278483-9480-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790321392278483-9480-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.66.62:33937
--webserver_interface=127.9.66.62
--webserver_port=0
--builtin_ntp_servers=127.9.66.20:35335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.66.62:33937
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 07:34:12.023694 28092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:34:12.023911 28092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:34:12.023933 28092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:34:12.025105 28092 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:34:12.025153 28092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:34:12.025167 28092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:34:12.025180 28092 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:34:12.025238 28092 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:34:12.026631 28092 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:35335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790321392278483-9480-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790321392278483-9480-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.66.62:33937
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.66.62:33937
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.66.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28092
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790321392278483-9480-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:34:12.026898 28092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:34:12.027242 28092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 07:34:12.032267 28092 server_base.cc:1034] running on GCE node
W20260925 07:34:12.032255 28097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:34:12.032215 28098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:34:12.032199 28100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:34:12.032789 28092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:34:12.033131 28092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:34:12.034283 28092 hybrid_clock.cc:648] HybridClock initialized: now 1790321652034277 us; error 38 us; skew 500 ppm
I20260925 07:34:12.035535 28092 webserver.cc:460] Webserver started at http://127.9.66.62:43151/ using document root <none> and password file <none>
I20260925 07:34:12.035778 28092 fs_manager.cc:359] Metadata directory not provided
I20260925 07:34:12.035845 28092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:34:12.036005 28092 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:34:12.037067 28092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790321392278483-9480-0/minicluster-data/master-0/data/instance:
uuid: "7a8cb8ed12f344cb8960dfebbbbae697"
format_stamp: "Formatted at 2026-09-25 07:34:12 on dist-test-slave-57fn"
server_key: "0ab6f7d410d4bd301ee8b24f27c19a38"
server_key_iv: "320818dff19ddc43bcf55d1dac90f9be"
server_key_version: "encryptionkey@0"
I20260925 07:34:12.037418 28092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790321392278483-9480-0/minicluster-data/master-0/wal/instance:
uuid: "7a8cb8ed12f344cb8960dfebbbbae697"
format_stamp: "Formatted at 2026-09-25 07:34:12 on dist-test-slave-57fn"
server_key: "0ab6f7d410d4bd301ee8b24f27c19a38"
server_key_iv: "320818dff19ddc43bcf55d1dac90f9be"
server_key_version: "encryptionkey@0"
I20260925 07:34:12.038724 28092 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 07:34:12.039402 28106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:34:12.039574 28092 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 07:34:12.039661 28092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790321392278483-9480-0/minicluster-data/master-0/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790321392278483-9480-0/minicluster-data/master-0/wal
uuid: "7a8cb8ed12f344cb8960dfebbbbae697"
format_stamp: "Formatted at 2026-09-25 07:34:12 on dist-test-slave-57fn"
server_key: "0ab6f7d410d4bd301ee8b24f27c19a38"
server_key_iv: "320818dff19ddc43bcf55d1dac90f9be"
server_key_version: "encryptionkey@0"
I20260925 07:34:12.039736 28092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790321392278483-9480-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790321392278483-9480-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790321392278483-9480-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:34:12.051169 28092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:34:12.051431 28092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:34:12.051576 28092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:34:12.055205 28092 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.62:33937
I20260925 07:34:12.055320 28158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.62:33937 every 8 connection(s)
I20260925 07:34:12.055619 28092 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
I20260925 07:34:12.056612 28159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:34:12.056932 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 28092
I20260925 07:34:12.057056 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790321392278483-9480-0/minicluster-data/master-0/wal/instance
I20260925 07:34:12.057271 9480 external_mini_cluster.cc:1428] Setting key 209cddfe3afe971a34c298650debb012
I20260925 07:34:12.059958 28159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a8cb8ed12f344cb8960dfebbbbae697: Bootstrap starting.
I20260925 07:34:12.060644 28159 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a8cb8ed12f344cb8960dfebbbbae697: Neither blocks nor log segments found. Creating new log.
I20260925 07:34:12.060971 28159 log.cc:824] T 00000000000000000000000000000000 P 7a8cb8ed12f344cb8960dfebbbbae697: Log is configured to *not* fsync() on all Append() calls
I20260925 07:34:12.061825 28159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a8cb8ed12f344cb8960dfebbbbae697: No bootstrap required, opened a new log
I20260925 07:34:12.062798 28162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:12.057802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35022 (local address 127.9.66.62:33937)
0925 07:34:12.058010 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:12.058024 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:12.058046 (+ 22us) server_negotiation.cc:407] Connection header received
0925 07:34:12.058457 (+ 411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:12.058463 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:12.058517 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:12.058621 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:34:12.058919 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:12.059738 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:12.060812 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:12.061265 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:12.061348 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:12.062135 (+ 787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:34:12.062139 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:34:12.062139 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 07:34:12.062323 (+ 184us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:34:12.062363 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 07:34:12.062379 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:34:12.062454 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:12.062510 (+ 56us) server_negotiation.cc:299] Negotiation successful
0925 07:34:12.062581 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":103,"threads_started":1}
I20260925 07:34:12.063829 28159 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a8cb8ed12f344cb8960dfebbbbae697 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a8cb8ed12f344cb8960dfebbbbae697" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 33937 } }
I20260925 07:34:12.063961 28159 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7a8cb8ed12f344cb8960dfebbbbae697 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:34:12.063983 28159 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a8cb8ed12f344cb8960dfebbbbae697 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8cb8ed12f344cb8960dfebbbbae697, State: Initialized, Role: FOLLOWER
I20260925 07:34:12.064114 28159 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a8cb8ed12f344cb8960dfebbbbae697 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a8cb8ed12f344cb8960dfebbbbae697" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 33937 } }
I20260925 07:34:12.064189 28159 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7a8cb8ed12f344cb8960dfebbbbae697 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 07:34:12.064226 28159 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7a8cb8ed12f344cb8960dfebbbbae697 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 07:34:12.064270 28159 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a8cb8ed12f344cb8960dfebbbbae697 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:34:12.064724 28159 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7a8cb8ed12f344cb8960dfebbbbae697 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a8cb8ed12f344cb8960dfebbbbae697" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 33937 } }
I20260925 07:34:12.064798 28159 leader_election.cc:302] T 00000000000000000000000000000000 P 7a8cb8ed12f344cb8960dfebbbbae697 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7a8cb8ed12f344cb8960dfebbbbae697; no voters:
I20260925 07:34:12.064971 28159 leader_election.cc:288] T 00000000000000000000000000000000 P 7a8cb8ed12f344cb8960dfebbbbae697 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 07:34:12.065034 28164 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7a8cb8ed12f344cb8960dfebbbbae697 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:34:12.065230 28164 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7a8cb8ed12f344cb8960dfebbbbae697 [term 1 LEADER]: Becoming Leader. State: Replica: 7a8cb8ed12f344cb8960dfebbbbae697, State: Running, Role: LEADER
I20260925 07:34:12.065243 28159 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a8cb8ed12f344cb8960dfebbbbae697 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:34:12.065325 28164 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a8cb8ed12f344cb8960dfebbbbae697 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a8cb8ed12f344cb8960dfebbbbae697" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 33937 } }
I20260925 07:34:12.065748 28166 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a8cb8ed12f344cb8960dfebbbbae697 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a8cb8ed12f344cb8960dfebbbbae697. Latest consensus state: current_term: 1 leader_uuid: "7a8cb8ed12f344cb8960dfebbbbae697" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a8cb8ed12f344cb8960dfebbbbae697" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 33937 } } }
I20260925 07:34:12.065821 28166 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a8cb8ed12f344cb8960dfebbbbae697 [sys.catalog]: This master's current role is: LEADER
I20260925 07:34:12.065821 28165 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a8cb8ed12f344cb8960dfebbbbae697 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7a8cb8ed12f344cb8960dfebbbbae697" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a8cb8ed12f344cb8960dfebbbbae697" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 33937 } } }
I20260925 07:34:12.066036 28165 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a8cb8ed12f344cb8960dfebbbbae697 [sys.catalog]: This master's current role is: LEADER
I20260925 07:34:12.066154 28173 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 07:34:12.066606 28173 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 07:34:12.070447 28173 catalog_manager.cc:1347] Generated new cluster ID: 2ad9bd603710446c98dd7cc72575eac3
I20260925 07:34:12.070509 28173 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 07:34:12.083681 28173 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 07:34:12.084329 28173 catalog_manager.cc:1504] Loading token signing keys...
I20260925 07:34:12.093540 28173 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7a8cb8ed12f344cb8960dfebbbbae697: Generated new TSK 0
I20260925 07:34:12.093746 28173 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 07:34:12.098639 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790321392278483-9480-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790321392278483-9480-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790321392278483-9480-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790321392278483-9480-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.66.1:0
--local_ip_for_outbound_sockets=127.9.66.1
--webserver_interface=127.9.66.1
--webserver_port=0
--tserver_master_addrs=127.9.66.62:33937
--builtin_ntp_servers=127.9.66.20:35335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 07:34:12.176955 28183 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:34:12.177135 28183 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:34:12.177156 28183 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:34:12.178303 28183 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:34:12.178340 28183 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:34:12.178383 28183 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.66.1
I20260925 07:34:12.179672 28183 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:35335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790321392278483-9480-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790321392278483-9480-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.66.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790321392278483-9480-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.66.1
--webserver_port=0
--tserver_master_addrs=127.9.66.62:33937
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28183
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.66.1
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790321392278483-9480-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:34:12.179876 28183 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:34:12.180159 28183 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:34:12.184922 28189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:34:12.184933 28188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:34:12.185211 28183 server_base.cc:1034] running on GCE node
W20260925 07:34:12.184952 28191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:34:12.185446 28183 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:34:12.185714 28183 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:34:12.186866 28183 hybrid_clock.cc:648] HybridClock initialized: now 1790321652186853 us; error 30 us; skew 500 ppm
I20260925 07:34:12.192706 28183 webserver.cc:460] Webserver started at http://127.9.66.1:46577/ using document root <none> and password file <none>
I20260925 07:34:12.192955 28183 fs_manager.cc:359] Metadata directory not provided
I20260925 07:34:12.193001 28183 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:34:12.193117 28183 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:34:12.194152 28183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790321392278483-9480-0/minicluster-data/ts-0/data/instance:
uuid: "c9a984dd833f49d2834d9508b746d9a8"
format_stamp: "Formatted at 2026-09-25 07:34:12 on dist-test-slave-57fn"
server_key: "6b3b41e8d6d618762d2a1fc7f4a949e0"
server_key_iv: "d1598fedeaea62650a451cb5a11c0b0a"
server_key_version: "encryptionkey@0"
I20260925 07:34:12.194468 28183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790321392278483-9480-0/minicluster-data/ts-0/wal/instance:
uuid: "c9a984dd833f49d2834d9508b746d9a8"
format_stamp: "Formatted at 2026-09-25 07:34:12 on dist-test-slave-57fn"
server_key: "6b3b41e8d6d618762d2a1fc7f4a949e0"
server_key_iv: "d1598fedeaea62650a451cb5a11c0b0a"
server_key_version: "encryptionkey@0"
I20260925 07:34:12.195675 28183 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 07:34:12.196318 28197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:34:12.196494 28183 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 07:34:12.196563 28183 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790321392278483-9480-0/minicluster-data/ts-0/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790321392278483-9480-0/minicluster-data/ts-0/wal
uuid: "c9a984dd833f49d2834d9508b746d9a8"
format_stamp: "Formatted at 2026-09-25 07:34:12 on dist-test-slave-57fn"
server_key: "6b3b41e8d6d618762d2a1fc7f4a949e0"
server_key_iv: "d1598fedeaea62650a451cb5a11c0b0a"
server_key_version: "encryptionkey@0"
I20260925 07:34:12.196614 28183 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790321392278483-9480-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790321392278483-9480-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790321392278483-9480-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:34:12.207558 28183 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:34:12.207764 28183 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:34:12.207866 28183 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:34:12.208019 28183 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:34:12.208312 28183 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:34:12.208370 28183 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:34:12.208420 28183 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:34:12.208446 28183 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:34:12.213882 28183 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.1:32839
I20260925 07:34:12.213958 28310 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.1:32839 every 8 connection(s)
I20260925 07:34:12.214263 28183 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790321392278483-9480-0/minicluster-data/ts-0/data/info.pb
I20260925 07:34:12.219410 28162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:12.214923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.1:47827 (local address 127.9.66.62:33937)
0925 07:34:12.214987 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:12.214989 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:12.215197 (+ 208us) server_negotiation.cc:407] Connection header received
0925 07:34:12.215710 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:12.215711 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:12.215727 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:12.215780 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:34:12.216449 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:12.216980 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:12.218048 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:12.218286 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:12.218361 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:12.219147 (+ 786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:34:12.219148 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:34:12.219148 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 07:34:12.219153 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:34:12.219165 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 07:34:12.219169 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:34:12.219214 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:12.219308 (+ 94us) server_negotiation.cc:299] Negotiation successful
0925 07:34:12.219335 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 07:34:12.219607 28313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:12.214910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:33937 (local address 127.9.66.1:47827)
0925 07:34:12.215066 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 07:34:12.215119 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 07:34:12.215602 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:34:12.215839 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:34:12.215842 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:34:12.215851 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:34:12.216350 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:12.216354 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:12.217076 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:12.217077 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:12.217960 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:12.217962 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:12.218378 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:12.218379 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:12.218495 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:12.219001 (+ 506us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 07:34:12.219002 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:34:12.219027 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 07:34:12.219048 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:34:12.219214 (+ 166us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:34:12.219219 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:34:12.219222 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 07:34:12.219297 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 07:34:12.219400 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":57,"threads_started":1}
I20260925 07:34:12.220112 28311 heartbeater.cc:347] Connected to a master server at 127.9.66.62:33937
I20260925 07:34:12.220214 28311 heartbeater.cc:464] Registering TS with master...
I20260925 07:34:12.220417 28311 heartbeater.cc:511] Master 127.9.66.62:33937 requested a full tablet report, sending...
I20260925 07:34:12.220924 28123 ts_manager.cc:194] Registered new tserver with Master: c9a984dd833f49d2834d9508b746d9a8 (127.9.66.1:32839)
I20260925 07:34:12.221550 28123 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.66.1:47827
I20260925 07:34:12.223567 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 28183
I20260925 07:34:12.223690 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790321392278483-9480-0/minicluster-data/ts-0/wal/instance
I20260925 07:34:12.223852 9480 external_mini_cluster.cc:1428] Setting key 41116bc2fcfc325c070035edde8363ca
I20260925 07:34:12.224630 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790321392278483-9480-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790321392278483-9480-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790321392278483-9480-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790321392278483-9480-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.66.2:0
--local_ip_for_outbound_sockets=127.9.66.2
--webserver_interface=127.9.66.2
--webserver_port=0
--tserver_master_addrs=127.9.66.62:33937
--builtin_ntp_servers=127.9.66.20:35335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 07:34:12.302932 28314 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:34:12.303143 28314 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:34:12.303172 28314 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:34:12.304165 28314 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:34:12.304221 28314 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:34:12.304257 28314 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.66.2
I20260925 07:34:12.305466 28314 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:35335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790321392278483-9480-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790321392278483-9480-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.66.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790321392278483-9480-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.66.2
--webserver_port=0
--tserver_master_addrs=127.9.66.62:33937
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28314
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.66.2
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790321392278483-9480-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:34:12.305747 28314 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:34:12.306017 28314 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:34:12.310596 28322 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:34:12.310606 28319 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:34:12.310621 28314 server_base.cc:1034] running on GCE node
W20260925 07:34:12.310609 28320 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:34:12.310977 28314 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:34:12.311188 28314 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:34:12.312382 28314 hybrid_clock.cc:648] HybridClock initialized: now 1790321652312372 us; error 28 us; skew 500 ppm
I20260925 07:34:12.313726 28314 webserver.cc:460] Webserver started at http://127.9.66.2:33119/ using document root <none> and password file <none>
I20260925 07:34:12.313947 28314 fs_manager.cc:359] Metadata directory not provided
I20260925 07:34:12.314013 28314 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:34:12.314136 28314 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:34:12.315183 28314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790321392278483-9480-0/minicluster-data/ts-1/data/instance:
uuid: "44a28723926d41438e975ac33c4d2ab2"
format_stamp: "Formatted at 2026-09-25 07:34:12 on dist-test-slave-57fn"
server_key: "15880b4167e9ea4f7b8e16de2fbdab60"
server_key_iv: "730df5cf4fe5a19486879642309d1735"
server_key_version: "encryptionkey@0"
I20260925 07:34:12.315546 28314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790321392278483-9480-0/minicluster-data/ts-1/wal/instance:
uuid: "44a28723926d41438e975ac33c4d2ab2"
format_stamp: "Formatted at 2026-09-25 07:34:12 on dist-test-slave-57fn"
server_key: "15880b4167e9ea4f7b8e16de2fbdab60"
server_key_iv: "730df5cf4fe5a19486879642309d1735"
server_key_version: "encryptionkey@0"
I20260925 07:34:12.316838 28314 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 07:34:12.317693 28328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:34:12.317898 28314 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:34:12.318011 28314 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790321392278483-9480-0/minicluster-data/ts-1/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790321392278483-9480-0/minicluster-data/ts-1/wal
uuid: "44a28723926d41438e975ac33c4d2ab2"
format_stamp: "Formatted at 2026-09-25 07:34:12 on dist-test-slave-57fn"
server_key: "15880b4167e9ea4f7b8e16de2fbdab60"
server_key_iv: "730df5cf4fe5a19486879642309d1735"
server_key_version: "encryptionkey@0"
I20260925 07:34:12.318095 28314 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790321392278483-9480-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790321392278483-9480-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790321392278483-9480-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:34:12.332541 28314 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:34:12.332792 28314 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:34:12.332923 28314 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:34:12.333118 28314 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:34:12.333411 28314 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:34:12.333461 28314 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:34:12.333525 28314 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:34:12.333552 28314 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:34:12.339459 28314 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.2:37417
I20260925 07:34:12.339497 28441 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.2:37417 every 8 connection(s)
I20260925 07:34:12.339807 28314 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790321392278483-9480-0/minicluster-data/ts-1/data/info.pb
I20260925 07:34:12.344879 28162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:12.340495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.2:33207 (local address 127.9.66.62:33937)
0925 07:34:12.340602 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:12.340604 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:12.340784 (+ 180us) server_negotiation.cc:407] Connection header received
0925 07:34:12.341278 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:12.341279 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:12.341291 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:12.341328 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:34:12.342085 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:12.342594 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:12.343550 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:12.343752 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:12.343823 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:12.344545 (+ 722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:34:12.344546 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:34:12.344547 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 07:34:12.344551 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:34:12.344583 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 07:34:12.344586 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:34:12.344670 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:12.344755 (+ 85us) server_negotiation.cc:299] Negotiation successful
0925 07:34:12.344779 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 07:34:12.345129 28444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:12.340450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:33937 (local address 127.9.66.2:33207)
0925 07:34:12.340676 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0925 07:34:12.340706 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0925 07:34:12.341176 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:34:12.341415 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:34:12.341418 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:34:12.341425 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:34:12.341988 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:12.341993 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:12.342688 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:12.342689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:12.343453 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:12.343455 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:12.343843 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:12.343844 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:12.343968 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:12.344422 (+ 454us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 07:34:12.344422 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:34:12.344446 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 07:34:12.344465 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:34:12.344672 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:34:12.344677 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:34:12.344680 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 07:34:12.344756 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 07:34:12.344865 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":93,"threads_started":1}
I20260925 07:34:12.345646 28442 heartbeater.cc:347] Connected to a master server at 127.9.66.62:33937
I20260925 07:34:12.345716 28442 heartbeater.cc:464] Registering TS with master...
I20260925 07:34:12.345901 28442 heartbeater.cc:511] Master 127.9.66.62:33937 requested a full tablet report, sending...
I20260925 07:34:12.346261 28123 ts_manager.cc:194] Registered new tserver with Master: 44a28723926d41438e975ac33c4d2ab2 (127.9.66.2:37417)
I20260925 07:34:12.346632 28123 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.66.2:33207
I20260925 07:34:12.349356 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 28314
I20260925 07:34:12.349438 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790321392278483-9480-0/minicluster-data/ts-1/wal/instance
I20260925 07:34:12.349638 9480 external_mini_cluster.cc:1428] Setting key 3fa2216b4dc3c06551a43cf40597814a
I20260925 07:34:12.350430 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790321392278483-9480-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790321392278483-9480-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790321392278483-9480-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790321392278483-9480-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.66.3:0
--local_ip_for_outbound_sockets=127.9.66.3
--webserver_interface=127.9.66.3
--webserver_port=0
--tserver_master_addrs=127.9.66.62:33937
--builtin_ntp_servers=127.9.66.20:35335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 07:34:12.429906 28445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:34:12.430119 28445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:34:12.430152 28445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:34:12.431284 28445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:34:12.431345 28445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:34:12.431448 28445 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.66.3
I20260925 07:34:12.432750 28445 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:35335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790321392278483-9480-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790321392278483-9480-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.66.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790321392278483-9480-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.66.3
--webserver_port=0
--tserver_master_addrs=127.9.66.62:33937
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28445
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.66.3
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790321392278483-9480-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:34:12.432986 28445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:34:12.433257 28445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:34:12.437790 28450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:34:12.437786 28451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:34:12.437827 28445 server_base.cc:1034] running on GCE node
W20260925 07:34:12.437824 28453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:34:12.438220 28445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:34:12.438490 28445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:34:12.439641 28445 hybrid_clock.cc:648] HybridClock initialized: now 1790321652439629 us; error 36 us; skew 500 ppm
I20260925 07:34:12.440925 28445 webserver.cc:460] Webserver started at http://127.9.66.3:34049/ using document root <none> and password file <none>
I20260925 07:34:12.441179 28445 fs_manager.cc:359] Metadata directory not provided
I20260925 07:34:12.441244 28445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:34:12.441377 28445 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:34:12.442469 28445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790321392278483-9480-0/minicluster-data/ts-2/data/instance:
uuid: "4cc419f7460f4f6ca46f15e4865aa55d"
format_stamp: "Formatted at 2026-09-25 07:34:12 on dist-test-slave-57fn"
server_key: "7d5031d55b9d93b9abc4f6ef94ced92c"
server_key_iv: "18dc79beaa9d47b086218732985859a4"
server_key_version: "encryptionkey@0"
I20260925 07:34:12.442786 28445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790321392278483-9480-0/minicluster-data/ts-2/wal/instance:
uuid: "4cc419f7460f4f6ca46f15e4865aa55d"
format_stamp: "Formatted at 2026-09-25 07:34:12 on dist-test-slave-57fn"
server_key: "7d5031d55b9d93b9abc4f6ef94ced92c"
server_key_iv: "18dc79beaa9d47b086218732985859a4"
server_key_version: "encryptionkey@0"
I20260925 07:34:12.444037 28445 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 07:34:12.444726 28459 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:34:12.444908 28445 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 07:34:12.445017 28445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790321392278483-9480-0/minicluster-data/ts-2/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790321392278483-9480-0/minicluster-data/ts-2/wal
uuid: "4cc419f7460f4f6ca46f15e4865aa55d"
format_stamp: "Formatted at 2026-09-25 07:34:12 on dist-test-slave-57fn"
server_key: "7d5031d55b9d93b9abc4f6ef94ced92c"
server_key_iv: "18dc79beaa9d47b086218732985859a4"
server_key_version: "encryptionkey@0"
I20260925 07:34:12.445093 28445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790321392278483-9480-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790321392278483-9480-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790321392278483-9480-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:34:12.456581 28445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:34:12.456874 28445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:34:12.457018 28445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:34:12.457217 28445 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:34:12.457599 28445 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:34:12.457651 28445 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:34:12.457693 28445 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:34:12.457719 28445 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:34:12.463198 28445 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.3:44513
I20260925 07:34:12.463239 28572 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.3:44513 every 8 connection(s)
I20260925 07:34:12.463563 28445 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790321392278483-9480-0/minicluster-data/ts-2/data/info.pb
I20260925 07:34:12.464711 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 28445
I20260925 07:34:12.464828 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790321392278483-9480-0/minicluster-data/ts-2/wal/instance
I20260925 07:34:12.465049 9480 external_mini_cluster.cc:1428] Setting key 577a1bff71b7b99381eedcc5bee4f306
I20260925 07:34:12.468895 28162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:12.464240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.3:43631 (local address 127.9.66.62:33937)
0925 07:34:12.464290 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:12.464292 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:12.464482 (+ 190us) server_negotiation.cc:407] Connection header received
0925 07:34:12.465128 (+ 646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:12.465130 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:12.465145 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:12.465219 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:34:12.466103 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:12.466573 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:12.467602 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:12.467788 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:12.467850 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:12.468655 (+ 805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:34:12.468657 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:34:12.468657 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 07:34:12.468662 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:34:12.468678 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 07:34:12.468681 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:34:12.468740 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:12.468816 (+ 76us) server_negotiation.cc:299] Negotiation successful
0925 07:34:12.468838 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 07:34:12.469095 28575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:12.464202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:33937 (local address 127.9.66.3:43631)
0925 07:34:12.464374 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 07:34:12.464414 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 07:34:12.465022 (+ 608us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:34:12.465287 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:34:12.465291 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:34:12.465301 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:34:12.466030 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:12.466036 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:12.466649 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:12.466650 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:12.467504 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:12.467507 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:12.467865 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:12.467866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:12.467965 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:12.468511 (+ 546us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 07:34:12.468511 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:34:12.468537 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 07:34:12.468556 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:34:12.468749 (+ 193us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:34:12.468753 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:34:12.468755 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 07:34:12.468803 (+ 48us) client_negotiation.cc:240] Negotiation successful
0925 07:34:12.468872 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":89,"threads_started":1}
I20260925 07:34:12.469720 28573 heartbeater.cc:347] Connected to a master server at 127.9.66.62:33937
I20260925 07:34:12.469805 28573 heartbeater.cc:464] Registering TS with master...
I20260925 07:34:12.469944 28573 heartbeater.cc:511] Master 127.9.66.62:33937 requested a full tablet report, sending...
I20260925 07:34:12.470287 28123 ts_manager.cc:194] Registered new tserver with Master: 4cc419f7460f4f6ca46f15e4865aa55d (127.9.66.3:44513)
I20260925 07:34:12.470626 28123 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.66.3:43631
I20260925 07:34:12.476786 9480 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 07:34:12.480861 28162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:12.478151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35032 (local address 127.9.66.62:33937)
0925 07:34:12.478228 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:12.478230 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:12.478268 (+ 38us) server_negotiation.cc:407] Connection header received
0925 07:34:12.478328 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:12.478329 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:12.478336 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:12.478373 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:34:12.478549 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:12.478977 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:12.479776 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:12.479948 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:12.480001 (+ 53us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:12.480581 (+ 580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:34:12.480582 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:34:12.480582 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 07:34:12.480586 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:34:12.480598 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 07:34:12.480602 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:34:12.480674 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:12.480733 (+ 59us) server_negotiation.cc:299] Negotiation successful
0925 07:34:12.480744 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 07:34:12.482595 28123 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35032:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 07:34:12.484095 28123 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 07:34:12.490357 28585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:12.486481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.1:32839 (local address 127.0.0.1:36252)
0925 07:34:12.486777 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0925 07:34:12.486787 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 07:34:12.486831 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:34:12.487153 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:34:12.487154 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:34:12.487155 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:34:12.487190 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:12.487192 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:12.488074 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:12.488075 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:12.489313 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:12.489315 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:12.490076 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:12.490077 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:12.490148 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:12.490159 (+ 11us) client_negotiation.cc:769] Sending connection context
0925 07:34:12.490225 (+ 66us) client_negotiation.cc:240] Negotiation successful
0925 07:34:12.490277 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":77,"threads_started":1}
I20260925 07:34:12.490443 28584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:12.486449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47948 (local address 127.9.66.2:37417)
0925 07:34:12.486639 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:12.486644 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:12.486655 (+ 11us) server_negotiation.cc:407] Connection header received
0925 07:34:12.486697 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:12.486699 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:12.486715 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:12.486990 (+ 275us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:34:12.487054 (+ 64us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:12.488197 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:12.489405 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:12.490117 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:12.490177 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:12.490250 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:12.490325 (+ 75us) server_negotiation.cc:299] Negotiation successful
0925 07:34:12.490362 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":103,"threads_started":1}
I20260925 07:34:12.490526 28583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:12.486354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48774 (local address 127.9.66.3:44513)
0925 07:34:12.486695 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:12.486700 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:12.486711 (+ 11us) server_negotiation.cc:407] Connection header received
0925 07:34:12.486821 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:12.486824 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:12.486838 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:12.487078 (+ 240us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:34:12.487157 (+ 79us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:12.488218 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:12.489251 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:12.490006 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:12.490078 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:12.490165 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:12.490401 (+ 236us) server_negotiation.cc:299] Negotiation successful
0925 07:34:12.490442 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":89,"threads_started":1}
I20260925 07:34:12.490559 28586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:12.486688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36252 (local address 127.9.66.1:32839)
0925 07:34:12.486968 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:12.486972 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:12.486981 (+ 9us) server_negotiation.cc:407] Connection header received
0925 07:34:12.487019 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:12.487022 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:12.487038 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:12.487267 (+ 229us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:34:12.487331 (+ 64us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:12.487991 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:12.489436 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:12.489980 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:12.490072 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:12.490161 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:12.490412 (+ 251us) server_negotiation.cc:299] Negotiation successful
0925 07:34:12.490458 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":84,"threads_started":1}
I20260925 07:34:12.490777 28581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:12.486281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.2:37417 (local address 127.0.0.1:47948)
0925 07:34:12.486442 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 07:34:12.486478 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 07:34:12.486538 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:34:12.486826 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:34:12.486828 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:34:12.486831 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:34:12.486899 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:12.486901 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:12.488286 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:12.488286 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:12.489315 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:12.489316 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:12.490208 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:12.490209 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:12.490261 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:12.490264 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 07:34:12.490314 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 07:34:12.490347 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":86,"threads_started":1}
I20260925 07:34:12.490839 28582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:12.486281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.3:44513 (local address 127.0.0.1:48774)
0925 07:34:12.486655 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0925 07:34:12.486666 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 07:34:12.486716 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:34:12.486931 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:34:12.486932 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:34:12.486934 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:34:12.486980 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:12.486981 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:12.488305 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:12.488306 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:12.489188 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:12.489191 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:12.490201 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:12.490202 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:12.490268 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:12.490273 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 07:34:12.490321 (+ 48us) client_negotiation.cc:240] Negotiation successful
0925 07:34:12.490362 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"mutex_wait_us":2,"thread_start_us":62,"threads_started":1}
I20260925 07:34:12.491371 28245 tablet_service.cc:1467] Processing CreateTablet for tablet 2087c6260d494750936e6d09905301be (DEFAULT_TABLE table=test-table [id=82f6d952afd54e74a9f4bb91511397b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:34:12.491446 28507 tablet_service.cc:1467] Processing CreateTablet for tablet 2087c6260d494750936e6d09905301be (DEFAULT_TABLE table=test-table [id=82f6d952afd54e74a9f4bb91511397b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:34:12.491706 28245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2087c6260d494750936e6d09905301be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:34:12.491743 28507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2087c6260d494750936e6d09905301be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:34:12.494282 28587 tablet_bootstrap.cc:492] T 2087c6260d494750936e6d09905301be P c9a984dd833f49d2834d9508b746d9a8: Bootstrap starting.
I20260925 07:34:12.494661 28588 tablet_bootstrap.cc:492] T 2087c6260d494750936e6d09905301be P 4cc419f7460f4f6ca46f15e4865aa55d: Bootstrap starting.
I20260925 07:34:12.494740 28376 tablet_service.cc:1467] Processing CreateTablet for tablet 2087c6260d494750936e6d09905301be (DEFAULT_TABLE table=test-table [id=82f6d952afd54e74a9f4bb91511397b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:34:12.494988 28587 tablet_bootstrap.cc:654] T 2087c6260d494750936e6d09905301be P c9a984dd833f49d2834d9508b746d9a8: Neither blocks nor log segments found. Creating new log.
I20260925 07:34:12.495095 28376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2087c6260d494750936e6d09905301be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:34:12.495304 28588 tablet_bootstrap.cc:654] T 2087c6260d494750936e6d09905301be P 4cc419f7460f4f6ca46f15e4865aa55d: Neither blocks nor log segments found. Creating new log.
I20260925 07:34:12.495302 28587 log.cc:824] T 2087c6260d494750936e6d09905301be P c9a984dd833f49d2834d9508b746d9a8: Log is configured to *not* fsync() on all Append() calls
I20260925 07:34:12.495657 28588 log.cc:824] T 2087c6260d494750936e6d09905301be P 4cc419f7460f4f6ca46f15e4865aa55d: Log is configured to *not* fsync() on all Append() calls
I20260925 07:34:12.496187 28587 tablet_bootstrap.cc:492] T 2087c6260d494750936e6d09905301be P c9a984dd833f49d2834d9508b746d9a8: No bootstrap required, opened a new log
I20260925 07:34:12.496263 28587 ts_tablet_manager.cc:1397] T 2087c6260d494750936e6d09905301be P c9a984dd833f49d2834d9508b746d9a8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 07:34:12.498399 28591 tablet_bootstrap.cc:492] T 2087c6260d494750936e6d09905301be P 44a28723926d41438e975ac33c4d2ab2: Bootstrap starting.
I20260925 07:34:12.498950 28591 tablet_bootstrap.cc:654] T 2087c6260d494750936e6d09905301be P 44a28723926d41438e975ac33c4d2ab2: Neither blocks nor log segments found. Creating new log.
I20260925 07:34:12.498939 28587 raft_consensus.cc:348] T 2087c6260d494750936e6d09905301be P c9a984dd833f49d2834d9508b746d9a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cc419f7460f4f6ca46f15e4865aa55d" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 44513 } } peers { permanent_uuid: "44a28723926d41438e975ac33c4d2ab2" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 37417 } } peers { permanent_uuid: "c9a984dd833f49d2834d9508b746d9a8" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 32839 } }
I20260925 07:34:12.499097 28587 raft_consensus.cc:374] T 2087c6260d494750936e6d09905301be P c9a984dd833f49d2834d9508b746d9a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:34:12.499140 28587 raft_consensus.cc:729] T 2087c6260d494750936e6d09905301be P c9a984dd833f49d2834d9508b746d9a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9a984dd833f49d2834d9508b746d9a8, State: Initialized, Role: FOLLOWER
I20260925 07:34:12.499274 28588 tablet_bootstrap.cc:492] T 2087c6260d494750936e6d09905301be P 4cc419f7460f4f6ca46f15e4865aa55d: No bootstrap required, opened a new log
I20260925 07:34:12.499275 28591 log.cc:824] T 2087c6260d494750936e6d09905301be P 44a28723926d41438e975ac33c4d2ab2: Log is configured to *not* fsync() on all Append() calls
I20260925 07:34:12.499274 28587 consensus_queue.cc:260] T 2087c6260d494750936e6d09905301be P c9a984dd833f49d2834d9508b746d9a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cc419f7460f4f6ca46f15e4865aa55d" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 44513 } } peers { permanent_uuid: "44a28723926d41438e975ac33c4d2ab2" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 37417 } } peers { permanent_uuid: "c9a984dd833f49d2834d9508b746d9a8" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 32839 } }
I20260925 07:34:12.499352 28588 ts_tablet_manager.cc:1397] T 2087c6260d494750936e6d09905301be P 4cc419f7460f4f6ca46f15e4865aa55d: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260925 07:34:12.499542 28587 ts_tablet_manager.cc:1428] T 2087c6260d494750936e6d09905301be P c9a984dd833f49d2834d9508b746d9a8: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260925 07:34:12.500100 28591 tablet_bootstrap.cc:492] T 2087c6260d494750936e6d09905301be P 44a28723926d41438e975ac33c4d2ab2: No bootstrap required, opened a new log
I20260925 07:34:12.500176 28591 ts_tablet_manager.cc:1397] T 2087c6260d494750936e6d09905301be P 44a28723926d41438e975ac33c4d2ab2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 07:34:12.502000 28588 raft_consensus.cc:348] T 2087c6260d494750936e6d09905301be P 4cc419f7460f4f6ca46f15e4865aa55d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cc419f7460f4f6ca46f15e4865aa55d" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 44513 } } peers { permanent_uuid: "44a28723926d41438e975ac33c4d2ab2" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 37417 } } peers { permanent_uuid: "c9a984dd833f49d2834d9508b746d9a8" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 32839 } }
I20260925 07:34:12.502148 28588 raft_consensus.cc:374] T 2087c6260d494750936e6d09905301be P 4cc419f7460f4f6ca46f15e4865aa55d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:34:12.502187 28588 raft_consensus.cc:729] T 2087c6260d494750936e6d09905301be P 4cc419f7460f4f6ca46f15e4865aa55d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cc419f7460f4f6ca46f15e4865aa55d, State: Initialized, Role: FOLLOWER
I20260925 07:34:12.502305 28588 consensus_queue.cc:260] T 2087c6260d494750936e6d09905301be P 4cc419f7460f4f6ca46f15e4865aa55d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cc419f7460f4f6ca46f15e4865aa55d" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 44513 } } peers { permanent_uuid: "44a28723926d41438e975ac33c4d2ab2" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 37417 } } peers { permanent_uuid: "c9a984dd833f49d2834d9508b746d9a8" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 32839 } }
I20260925 07:34:12.502552 28588 ts_tablet_manager.cc:1428] T 2087c6260d494750936e6d09905301be P 4cc419f7460f4f6ca46f15e4865aa55d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 07:34:12.502703 28573 heartbeater.cc:503] Master 127.9.66.62:33937 was elected leader, sending a full tablet report...
I20260925 07:34:12.502610 28591 raft_consensus.cc:348] T 2087c6260d494750936e6d09905301be P 44a28723926d41438e975ac33c4d2ab2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cc419f7460f4f6ca46f15e4865aa55d" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 44513 } } peers { permanent_uuid: "44a28723926d41438e975ac33c4d2ab2" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 37417 } } peers { permanent_uuid: "c9a984dd833f49d2834d9508b746d9a8" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 32839 } }
I20260925 07:34:12.502741 28591 raft_consensus.cc:374] T 2087c6260d494750936e6d09905301be P 44a28723926d41438e975ac33c4d2ab2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:34:12.502772 28591 raft_consensus.cc:729] T 2087c6260d494750936e6d09905301be P 44a28723926d41438e975ac33c4d2ab2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44a28723926d41438e975ac33c4d2ab2, State: Initialized, Role: FOLLOWER
I20260925 07:34:12.502904 28591 consensus_queue.cc:260] T 2087c6260d494750936e6d09905301be P 44a28723926d41438e975ac33c4d2ab2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cc419f7460f4f6ca46f15e4865aa55d" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 44513 } } peers { permanent_uuid: "44a28723926d41438e975ac33c4d2ab2" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 37417 } } peers { permanent_uuid: "c9a984dd833f49d2834d9508b746d9a8" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 32839 } }
I20260925 07:34:12.503175 28591 ts_tablet_manager.cc:1428] T 2087c6260d494750936e6d09905301be P 44a28723926d41438e975ac33c4d2ab2: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 07:34:12.503232 28442 heartbeater.cc:503] Master 127.9.66.62:33937 was elected leader, sending a full tablet report...
I20260925 07:34:12.504500 28311 heartbeater.cc:503] Master 127.9.66.62:33937 was elected leader, sending a full tablet report...
I20260925 07:34:12.511137 28595 raft_consensus.cc:482] T 2087c6260d494750936e6d09905301be P 44a28723926d41438e975ac33c4d2ab2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 07:34:12.511220 28595 raft_consensus.cc:504] T 2087c6260d494750936e6d09905301be P 44a28723926d41438e975ac33c4d2ab2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cc419f7460f4f6ca46f15e4865aa55d" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 44513 } } peers { permanent_uuid: "44a28723926d41438e975ac33c4d2ab2" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 37417 } } peers { permanent_uuid: "c9a984dd833f49d2834d9508b746d9a8" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 32839 } }
I20260925 07:34:12.511442 28595 leader_election.cc:288] T 2087c6260d494750936e6d09905301be P 44a28723926d41438e975ac33c4d2ab2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cc419f7460f4f6ca46f15e4865aa55d (127.9.66.3:44513), c9a984dd833f49d2834d9508b746d9a8 (127.9.66.1:32839)
I20260925 07:34:12.515395 28586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:12.511809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.2:47189 (local address 127.9.66.1:32839)
0925 07:34:12.511843 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:12.511845 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:12.512006 (+ 161us) server_negotiation.cc:407] Connection header received
0925 07:34:12.512035 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:12.512036 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:12.512044 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:12.512094 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:34:12.512217 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:12.512908 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:12.514421 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:12.514962 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:12.515039 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:12.515048 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:12.515285 (+ 237us) server_negotiation.cc:299] Negotiation successful
0925 07:34:12.515331 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 07:34:12.515384 28444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:12.511643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.3:44513 (local address 127.9.66.2:48187)
0925 07:34:12.511698 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0925 07:34:12.511711 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 07:34:12.511754 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:34:12.511970 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:34:12.511971 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:34:12.511972 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:34:12.512046 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:12.512048 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:12.512984 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:12.512985 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:12.514314 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:12.514318 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:12.515079 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:12.515080 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:12.515166 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:12.515171 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 07:34:12.515242 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 07:34:12.515290 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":36}
I20260925 07:34:12.515393 28596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:12.511769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.1:32839 (local address 127.9.66.2:47189)
0925 07:34:12.511907 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 07:34:12.511917 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 07:34:12.511985 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:34:12.512097 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:34:12.512098 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:34:12.512099 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:34:12.512153 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:12.512154 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:12.512984 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:12.512985 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:12.514315 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:12.514318 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:12.515079 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:12.515080 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:12.515166 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:12.515171 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 07:34:12.515242 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 07:34:12.515295 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"mutex_wait_us":59,"thread_start_us":100,"threads_started":1}
I20260925 07:34:12.515437 28583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:12.511695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.2:48187 (local address 127.9.66.3:44513)
0925 07:34:12.511732 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:12.511734 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:12.511743 (+ 9us) server_negotiation.cc:407] Connection header received
0925 07:34:12.511858 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:12.511859 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:12.511867 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:12.511932 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:34:12.512118 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:12.512885 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:12.514422 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:12.514962 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:12.515039 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:12.515048 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:12.515285 (+ 237us) server_negotiation.cc:299] Negotiation successful
0925 07:34:12.515331 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 07:34:12.515743 28527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2087c6260d494750936e6d09905301be" candidate_uuid: "44a28723926d41438e975ac33c4d2ab2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc419f7460f4f6ca46f15e4865aa55d" is_pre_election: true
I20260925 07:34:12.515761 28265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2087c6260d494750936e6d09905301be" candidate_uuid: "44a28723926d41438e975ac33c4d2ab2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9a984dd833f49d2834d9508b746d9a8" is_pre_election: true
I20260925 07:34:12.515913 28527 raft_consensus.cc:2454] T 2087c6260d494750936e6d09905301be P 4cc419f7460f4f6ca46f15e4865aa55d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44a28723926d41438e975ac33c4d2ab2 in term 0.
I20260925 07:34:12.515921 28265 raft_consensus.cc:2454] T 2087c6260d494750936e6d09905301be P c9a984dd833f49d2834d9508b746d9a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44a28723926d41438e975ac33c4d2ab2 in term 0.
I20260925 07:34:12.516116 28330 leader_election.cc:302] T 2087c6260d494750936e6d09905301be P 44a28723926d41438e975ac33c4d2ab2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44a28723926d41438e975ac33c4d2ab2, c9a984dd833f49d2834d9508b746d9a8; no voters:
I20260925 07:34:12.516273 28595 raft_consensus.cc:2781] T 2087c6260d494750936e6d09905301be P 44a28723926d41438e975ac33c4d2ab2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 07:34:12.516346 28595 raft_consensus.cc:482] T 2087c6260d494750936e6d09905301be P 44a28723926d41438e975ac33c4d2ab2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 07:34:12.516364 28595 raft_consensus.cc:3037] T 2087c6260d494750936e6d09905301be P 44a28723926d41438e975ac33c4d2ab2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:34:12.517196 28595 raft_consensus.cc:504] T 2087c6260d494750936e6d09905301be P 44a28723926d41438e975ac33c4d2ab2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cc419f7460f4f6ca46f15e4865aa55d" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 44513 } } peers { permanent_uuid: "44a28723926d41438e975ac33c4d2ab2" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 37417 } } peers { permanent_uuid: "c9a984dd833f49d2834d9508b746d9a8" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 32839 } }
I20260925 07:34:12.517328 28595 leader_election.cc:288] T 2087c6260d494750936e6d09905301be P 44a28723926d41438e975ac33c4d2ab2 [CANDIDATE]: Term 1 election: Requested vote from peers 4cc419f7460f4f6ca46f15e4865aa55d (127.9.66.3:44513), c9a984dd833f49d2834d9508b746d9a8 (127.9.66.1:32839)
I20260925 07:34:12.517493 28265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2087c6260d494750936e6d09905301be" candidate_uuid: "44a28723926d41438e975ac33c4d2ab2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9a984dd833f49d2834d9508b746d9a8"
I20260925 07:34:12.517546 28527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2087c6260d494750936e6d09905301be" candidate_uuid: "44a28723926d41438e975ac33c4d2ab2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cc419f7460f4f6ca46f15e4865aa55d"
I20260925 07:34:12.517592 28265 raft_consensus.cc:3037] T 2087c6260d494750936e6d09905301be P c9a984dd833f49d2834d9508b746d9a8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:34:12.517606 28527 raft_consensus.cc:3037] T 2087c6260d494750936e6d09905301be P 4cc419f7460f4f6ca46f15e4865aa55d [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:34:12.518451 28265 raft_consensus.cc:2454] T 2087c6260d494750936e6d09905301be P c9a984dd833f49d2834d9508b746d9a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44a28723926d41438e975ac33c4d2ab2 in term 1.
I20260925 07:34:12.518451 28527 raft_consensus.cc:2454] T 2087c6260d494750936e6d09905301be P 4cc419f7460f4f6ca46f15e4865aa55d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44a28723926d41438e975ac33c4d2ab2 in term 1.
I20260925 07:34:12.518654 28330 leader_election.cc:302] T 2087c6260d494750936e6d09905301be P 44a28723926d41438e975ac33c4d2ab2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44a28723926d41438e975ac33c4d2ab2, 4cc419f7460f4f6ca46f15e4865aa55d; no voters:
I20260925 07:34:12.518750 28595 raft_consensus.cc:2781] T 2087c6260d494750936e6d09905301be P 44a28723926d41438e975ac33c4d2ab2 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:34:12.518887 28595 raft_consensus.cc:686] T 2087c6260d494750936e6d09905301be P 44a28723926d41438e975ac33c4d2ab2 [term 1 LEADER]: Becoming Leader. State: Replica: 44a28723926d41438e975ac33c4d2ab2, State: Running, Role: LEADER
I20260925 07:34:12.518980 28595 consensus_queue.cc:237] T 2087c6260d494750936e6d09905301be P 44a28723926d41438e975ac33c4d2ab2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cc419f7460f4f6ca46f15e4865aa55d" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 44513 } } peers { permanent_uuid: "44a28723926d41438e975ac33c4d2ab2" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 37417 } } peers { permanent_uuid: "c9a984dd833f49d2834d9508b746d9a8" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 32839 } }
I20260925 07:34:12.519742 28123 catalog_manager.cc:5495] T 2087c6260d494750936e6d09905301be P 44a28723926d41438e975ac33c4d2ab2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 44a28723926d41438e975ac33c4d2ab2 (127.9.66.2). New cstate: current_term: 1 leader_uuid: "44a28723926d41438e975ac33c4d2ab2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cc419f7460f4f6ca46f15e4865aa55d" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 44513 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44a28723926d41438e975ac33c4d2ab2" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 37417 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9a984dd833f49d2834d9508b746d9a8" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 32839 } health_report { overall_health: UNKNOWN } } }
I20260925 07:34:12.529017 28584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:12.526723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47954 (local address 127.9.66.2:37417)
0925 07:34:12.526775 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:12.526777 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:12.526783 (+ 6us) server_negotiation.cc:407] Connection header received
0925 07:34:12.526817 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:12.526817 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:12.526832 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:12.526921 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 07:34:12.527060 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:12.527527 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:12.528280 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:12.528473 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:12.528555 (+ 82us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:12.528688 (+ 133us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 07:34:12.528770 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 07:34:12.528865 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:12.528900 (+ 35us) server_negotiation.cc:299] Negotiation successful
0925 07:34:12.528907 (+ 7us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 07:34:12.530651 28265 raft_consensus.cc:1261] T 2087c6260d494750936e6d09905301be P c9a984dd833f49d2834d9508b746d9a8 [term 1 FOLLOWER]: Refusing update from remote peer 44a28723926d41438e975ac33c4d2ab2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:34:12.530650 28527 raft_consensus.cc:1261] T 2087c6260d494750936e6d09905301be P 4cc419f7460f4f6ca46f15e4865aa55d [term 1 FOLLOWER]: Refusing update from remote peer 44a28723926d41438e975ac33c4d2ab2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:34:12.530969 28597 consensus_queue.cc:1035] T 2087c6260d494750936e6d09905301be P 44a28723926d41438e975ac33c4d2ab2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cc419f7460f4f6ca46f15e4865aa55d" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 44513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:34:12.531080 28595 consensus_queue.cc:1035] T 2087c6260d494750936e6d09905301be P 44a28723926d41438e975ac33c4d2ab2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9a984dd833f49d2834d9508b746d9a8" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 32839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:34:12.533677 28601 mvcc.cc:205] Tried to move back new op lower bound from 7333157488763211776 to 7333157488718405632. Current Snapshot: MvccSnapshot[applied={T|T < 7333157488763211776}]
I20260925 07:34:12.533780 28602 mvcc.cc:205] Tried to move back new op lower bound from 7333157488763211776 to 7333157488718405632. Current Snapshot: MvccSnapshot[applied={T|T < 7333157488763211776}]
I20260925 07:34:12.535171 28123 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35032:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 07:34:12.535257 28123 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35032:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 07:34:12.535950 28123 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7a8cb8ed12f344cb8960dfebbbbae697: Sending DeleteTablet for 3 replicas of tablet 2087c6260d494750936e6d09905301be
I20260925 07:34:12.536252 28376 tablet_service.cc:1514] Processing DeleteTablet for tablet 2087c6260d494750936e6d09905301be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:34:12 UTC) from {username='slave'} at 127.0.0.1:47948
I20260925 07:34:12.536306 28245 tablet_service.cc:1514] Processing DeleteTablet for tablet 2087c6260d494750936e6d09905301be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:34:12 UTC) from {username='slave'} at 127.0.0.1:36252
I20260925 07:34:12.536314 28507 tablet_service.cc:1514] Processing DeleteTablet for tablet 2087c6260d494750936e6d09905301be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:34:12 UTC) from {username='slave'} at 127.0.0.1:48774
I20260925 07:34:12.536470 28609 tablet_replica.cc:330] T 2087c6260d494750936e6d09905301be P 44a28723926d41438e975ac33c4d2ab2: stopping tablet replica
I20260925 07:34:12.536551 28609 raft_consensus.cc:2229] T 2087c6260d494750936e6d09905301be P 44a28723926d41438e975ac33c4d2ab2 [term 1 LEADER]: Raft consensus shutting down.
I20260925 07:34:12.536684 28609 raft_consensus.cc:2258] T 2087c6260d494750936e6d09905301be P 44a28723926d41438e975ac33c4d2ab2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:34:12.536791 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 28183
I20260925 07:34:12.536842 28610 tablet_replica.cc:330] T 2087c6260d494750936e6d09905301be P 4cc419f7460f4f6ca46f15e4865aa55d: stopping tablet replica
I20260925 07:34:12.536924 28610 raft_consensus.cc:2229] T 2087c6260d494750936e6d09905301be P 4cc419f7460f4f6ca46f15e4865aa55d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 07:34:12.537019 28610 raft_consensus.cc:2258] T 2087c6260d494750936e6d09905301be P 4cc419f7460f4f6ca46f15e4865aa55d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:34:12.537609 28610 ts_tablet_manager.cc:1905] T 2087c6260d494750936e6d09905301be P 4cc419f7460f4f6ca46f15e4865aa55d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 07:34:12.537716 28609 ts_tablet_manager.cc:1905] T 2087c6260d494750936e6d09905301be P 44a28723926d41438e975ac33c4d2ab2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 07:34:12.539384 28610 ts_tablet_manager.cc:1918] T 2087c6260d494750936e6d09905301be P 4cc419f7460f4f6ca46f15e4865aa55d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 07:34:12.539462 28610 log.cc:1196] T 2087c6260d494750936e6d09905301be P 4cc419f7460f4f6ca46f15e4865aa55d: Deleting WAL directory at /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790321392278483-9480-0/minicluster-data/ts-2/wal/wals/2087c6260d494750936e6d09905301be
I20260925 07:34:12.539594 28609 ts_tablet_manager.cc:1918] T 2087c6260d494750936e6d09905301be P 44a28723926d41438e975ac33c4d2ab2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 07:34:12.539707 28609 log.cc:1196] T 2087c6260d494750936e6d09905301be P 44a28723926d41438e975ac33c4d2ab2: Deleting WAL directory at /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790321392278483-9480-0/minicluster-data/ts-1/wal/wals/2087c6260d494750936e6d09905301be
I20260925 07:34:12.539722 28610 ts_tablet_manager.cc:1939] T 2087c6260d494750936e6d09905301be P 4cc419f7460f4f6ca46f15e4865aa55d: Deleting consensus metadata
I20260925 07:34:12.540045 28609 ts_tablet_manager.cc:1939] T 2087c6260d494750936e6d09905301be P 44a28723926d41438e975ac33c4d2ab2: Deleting consensus metadata
I20260925 07:34:12.540246 28108 catalog_manager.cc:4841] TS 4cc419f7460f4f6ca46f15e4865aa55d (127.9.66.3:44513): tablet 2087c6260d494750936e6d09905301be (table test-table [id=82f6d952afd54e74a9f4bb91511397b7]) successfully deleted
I20260925 07:34:12.540429 28110 catalog_manager.cc:4841] TS 44a28723926d41438e975ac33c4d2ab2 (127.9.66.2:37417): tablet 2087c6260d494750936e6d09905301be (table test-table [id=82f6d952afd54e74a9f4bb91511397b7]) successfully deleted
W20260925 07:34:12.544701 28108 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.66.1:32839 (error 108)
I20260925 07:34:12.544946 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 28314
I20260925 07:34:12.545500 28582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:12.545018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.1:32839 (local address 127.0.0.1:36264)
0925 07:34:12.545427 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0925 07:34:12.545476 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.1:32839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391}
W20260925 07:34:12.546370 28108 catalog_manager.cc:4584] TS c9a984dd833f49d2834d9508b746d9a8 (127.9.66.1:32839): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2087c6260d494750936e6d09905301be: Network error: Client connection negotiation failed: client connection to 127.9.66.1:32839: connect: Connection refused (error 111)
I20260925 07:34:12.550642 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 28445
I20260925 07:34:12.555528 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 28092
2026-09-25T07:34:12Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (638 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790321392278483-9480-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 07:34:12 dist-test-slave-57fn krb5kdc[28615](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 07:34:12 dist-test-slave-57fn krb5kdc[28615](info): set up 2 sockets
Sep 25 07:34:12 dist-test-slave-57fn krb5kdc[28615](info): commencing operation
krb5kdc: starting...
W20260925 07:34:13.569432 9480 mini_kdc.cc:121] Time spent starting KDC: real 0.995s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790321392278483-9480-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790321392278483-9480-0/krb5kdc/test-user.keytab.
Sep 25 07:34:13 dist-test-slave-57fn krb5kdc[28615](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321653, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T07:34:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T07:34:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.66.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.66.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.66.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab.
Entry for principal kudu/127.9.66.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab.
I20260925 07:34:13.682055 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790321392278483-9480-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790321392278483-9480-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790321392278483-9480-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.66.62:34589
--webserver_interface=127.9.66.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab
--principal=kudu/127.9.66.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:43503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.66.62:34589
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:34:13.759469 28628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:34:13.759691 28628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:34:13.759728 28628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:34:13.760854 28628 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:34:13.760917 28628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:34:13.760942 28628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:34:13.760963 28628 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:34:13.760994 28628 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:34:13.762473 28628 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:43503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790321392278483-9480-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790321392278483-9480-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.66.62:34589
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab
--principal=kudu/127.9.66.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.66.62:34589
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.66.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28628
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790321392278483-9480-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:34:13.762722 28628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:34:13.762946 28628 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:34:13.767531 28633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:34:13.767539 28634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:34:13.767719 28628 server_base.cc:1034] running on GCE node
W20260925 07:34:13.767539 28636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:34:13.768030 28628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:34:13.768347 28628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:34:13.769560 28628 hybrid_clock.cc:648] HybridClock initialized: now 1790321653769547 us; error 45 us; skew 500 ppm
Sep 25 07:34:13 dist-test-slave-57fn krb5kdc[28615](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321653, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:34:13.772090 28628 init.cc:377] Logged in from keytab as kudu/127.9.66.62@KRBTEST.COM (short username kudu)
I20260925 07:34:13.772607 28628 webserver.cc:460] Webserver started at http://127.9.66.62:44925/ using document root <none> and password file <none>
I20260925 07:34:13.772845 28628 fs_manager.cc:359] Metadata directory not provided
I20260925 07:34:13.772912 28628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:34:13.773052 28628 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:34:13.774151 28628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790321392278483-9480-0/minicluster-data/master-0/data/instance:
uuid: "97b8026bd9544e9ba5e5ef1e6545169b"
format_stamp: "Formatted at 2026-09-25 07:34:13 on dist-test-slave-57fn"
server_key: "cbf78a4dd65f2872290ce877a3197f6d"
server_key_iv: "83fca219e7877f4922660c37e04975fc"
server_key_version: "encryptionkey@0"
I20260925 07:34:13.774528 28628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790321392278483-9480-0/minicluster-data/master-0/wal/instance:
uuid: "97b8026bd9544e9ba5e5ef1e6545169b"
format_stamp: "Formatted at 2026-09-25 07:34:13 on dist-test-slave-57fn"
server_key: "cbf78a4dd65f2872290ce877a3197f6d"
server_key_iv: "83fca219e7877f4922660c37e04975fc"
server_key_version: "encryptionkey@0"
I20260925 07:34:13.775827 28628 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 07:34:13.776602 28643 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:34:13.776780 28628 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:34:13.776902 28628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790321392278483-9480-0/minicluster-data/master-0/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790321392278483-9480-0/minicluster-data/master-0/wal
uuid: "97b8026bd9544e9ba5e5ef1e6545169b"
format_stamp: "Formatted at 2026-09-25 07:34:13 on dist-test-slave-57fn"
server_key: "cbf78a4dd65f2872290ce877a3197f6d"
server_key_iv: "83fca219e7877f4922660c37e04975fc"
server_key_version: "encryptionkey@0"
I20260925 07:34:13.776984 28628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790321392278483-9480-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790321392278483-9480-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790321392278483-9480-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:34:13.792608 28628 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:34:13.795020 28628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:34:13.795190 28628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:34:13.799470 28628 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.62:34589
I20260925 07:34:13.799502 28695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.62:34589 every 8 connection(s)
I20260925 07:34:13.799826 28628 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
I20260925 07:34:13.800645 28696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:34:13.802871 28696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97b8026bd9544e9ba5e5ef1e6545169b: Bootstrap starting.
I20260925 07:34:13.803362 28696 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 97b8026bd9544e9ba5e5ef1e6545169b: Neither blocks nor log segments found. Creating new log.
I20260925 07:34:13.803588 28696 log.cc:824] T 00000000000000000000000000000000 P 97b8026bd9544e9ba5e5ef1e6545169b: Log is configured to *not* fsync() on all Append() calls
I20260925 07:34:13.804222 28696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97b8026bd9544e9ba5e5ef1e6545169b: No bootstrap required, opened a new log
I20260925 07:34:13.805828 28696 raft_consensus.cc:348] T 00000000000000000000000000000000 P 97b8026bd9544e9ba5e5ef1e6545169b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97b8026bd9544e9ba5e5ef1e6545169b" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 34589 } }
I20260925 07:34:13.805971 28696 raft_consensus.cc:374] T 00000000000000000000000000000000 P 97b8026bd9544e9ba5e5ef1e6545169b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:34:13.805994 28696 raft_consensus.cc:729] T 00000000000000000000000000000000 P 97b8026bd9544e9ba5e5ef1e6545169b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97b8026bd9544e9ba5e5ef1e6545169b, State: Initialized, Role: FOLLOWER
I20260925 07:34:13.806110 28696 consensus_queue.cc:260] T 00000000000000000000000000000000 P 97b8026bd9544e9ba5e5ef1e6545169b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97b8026bd9544e9ba5e5ef1e6545169b" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 34589 } }
I20260925 07:34:13.806191 28696 raft_consensus.cc:388] T 00000000000000000000000000000000 P 97b8026bd9544e9ba5e5ef1e6545169b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 07:34:13.806226 28696 raft_consensus.cc:482] T 00000000000000000000000000000000 P 97b8026bd9544e9ba5e5ef1e6545169b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 07:34:13.806275 28696 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 97b8026bd9544e9ba5e5ef1e6545169b [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:34:13.806270 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 28628
I20260925 07:34:13.806365 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790321392278483-9480-0/minicluster-data/master-0/wal/instance
I20260925 07:34:13.806546 9480 external_mini_cluster.cc:1428] Setting key e1dda067fc7502580326c25d89335547
I20260925 07:34:13.806963 28696 raft_consensus.cc:504] T 00000000000000000000000000000000 P 97b8026bd9544e9ba5e5ef1e6545169b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97b8026bd9544e9ba5e5ef1e6545169b" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 34589 } }
I20260925 07:34:13.807089 28696 leader_election.cc:302] T 00000000000000000000000000000000 P 97b8026bd9544e9ba5e5ef1e6545169b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 97b8026bd9544e9ba5e5ef1e6545169b; no voters:
I20260925 07:34:13.807288 28696 leader_election.cc:288] T 00000000000000000000000000000000 P 97b8026bd9544e9ba5e5ef1e6545169b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 07:34:13.807314 28701 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 97b8026bd9544e9ba5e5ef1e6545169b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:34:13.807469 28701 raft_consensus.cc:686] T 00000000000000000000000000000000 P 97b8026bd9544e9ba5e5ef1e6545169b [term 1 LEADER]: Becoming Leader. State: Replica: 97b8026bd9544e9ba5e5ef1e6545169b, State: Running, Role: LEADER
I20260925 07:34:13.807534 28696 sys_catalog.cc:564] T 00000000000000000000000000000000 P 97b8026bd9544e9ba5e5ef1e6545169b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:34:13.807574 28701 consensus_queue.cc:237] T 00000000000000000000000000000000 P 97b8026bd9544e9ba5e5ef1e6545169b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97b8026bd9544e9ba5e5ef1e6545169b" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 34589 } }
I20260925 07:34:13.807893 28701 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97b8026bd9544e9ba5e5ef1e6545169b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 97b8026bd9544e9ba5e5ef1e6545169b. Latest consensus state: current_term: 1 leader_uuid: "97b8026bd9544e9ba5e5ef1e6545169b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97b8026bd9544e9ba5e5ef1e6545169b" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 34589 } } }
I20260925 07:34:13.807981 28701 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97b8026bd9544e9ba5e5ef1e6545169b [sys.catalog]: This master's current role is: LEADER
I20260925 07:34:13.808153 28702 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97b8026bd9544e9ba5e5ef1e6545169b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "97b8026bd9544e9ba5e5ef1e6545169b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97b8026bd9544e9ba5e5ef1e6545169b" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 34589 } } }
I20260925 07:34:13.808187 28707 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 07:34:13.808292 28702 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97b8026bd9544e9ba5e5ef1e6545169b [sys.catalog]: This master's current role is: LEADER
I20260925 07:34:13.808785 28707 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 07:34:13.810492 28707 catalog_manager.cc:1347] Generated new cluster ID: b764f157952b4a7fbf83a62aed96f339
I20260925 07:34:13.810544 28707 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 07:34:13 dist-test-slave-57fn krb5kdc[28615](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321653, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.66.62@KRBTEST.COM
I20260925 07:34:13.816975 28700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:13.807085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58752 (local address 127.9.66.62:34589)
0925 07:34:13.807291 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:13.807300 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:13.807334 (+ 34us) server_negotiation.cc:407] Connection header received
0925 07:34:13.807450 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:13.807458 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:13.807509 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:13.808521 (+ 1012us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:34:13.808640 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:13.810052 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:13.810964 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:13.811452 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:13.811576 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:13.813854 (+ 2278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:34:13.813857 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:34:13.813858 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:34:13.813869 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:34:13.815439 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:34:13.815804 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:34:13.815806 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:34:13.815807 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:34:13.815921 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:34:13.816137 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:34:13.816138 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:34:13.816138 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:34:13.816354 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:34:13.816522 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:13.816612 (+ 90us) server_negotiation.cc:299] Negotiation successful
0925 07:34:13.816747 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":97,"threads_started":1}
I20260925 07:34:13.821983 28707 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 07:34:13.822616 28707 catalog_manager.cc:1504] Loading token signing keys...
I20260925 07:34:13.826730 28707 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 97b8026bd9544e9ba5e5ef1e6545169b: Generated new TSK 0
I20260925 07:34:13.826910 28707 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.66.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.66.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.66.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.1.keytab.
Entry for principal kudu/127.9.66.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.1.keytab.
I20260925 07:34:13.859869 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790321392278483-9480-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790321392278483-9480-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790321392278483-9480-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790321392278483-9480-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.66.1:0
--local_ip_for_outbound_sockets=127.9.66.1
--webserver_interface=127.9.66.1
--webserver_port=0
--tserver_master_addrs=127.9.66.62:34589
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.1.keytab
--principal=kudu/127.9.66.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:43503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:34:13.956715 28722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:34:13.956914 28722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:34:13.956938 28722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:34:13.958168 28722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:34:13.958209 28722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:34:13.958235 28722 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.66.1
I20260925 07:34:13.959657 28722 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:43503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790321392278483-9480-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790321392278483-9480-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.1.keytab
--principal=kudu/127.9.66.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.66.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790321392278483-9480-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.66.1
--webserver_port=0
--tserver_master_addrs=127.9.66.62:34589
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28722
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.66.1
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790321392278483-9480-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:34:13.959888 28722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:34:13.960126 28722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:34:13.965898 28730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:34:13.965899 28728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:34:13.965899 28727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:34:13.966221 28722 server_base.cc:1034] running on GCE node
I20260925 07:34:13.966426 28722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:34:13.966704 28722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:34:13.967876 28722 hybrid_clock.cc:648] HybridClock initialized: now 1790321653967861 us; error 30 us; skew 500 ppm
Sep 25 07:34:13 dist-test-slave-57fn krb5kdc[28615](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321653, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:34:13.970566 28722 init.cc:377] Logged in from keytab as kudu/127.9.66.1@KRBTEST.COM (short username kudu)
I20260925 07:34:13.971065 28722 webserver.cc:460] Webserver started at http://127.9.66.1:37803/ using document root <none> and password file <none>
I20260925 07:34:13.971298 28722 fs_manager.cc:359] Metadata directory not provided
I20260925 07:34:13.971354 28722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:34:13.971535 28722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:34:13.972731 28722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790321392278483-9480-0/minicluster-data/ts-0/data/instance:
uuid: "27e86fb49a89468788317029f41d4fd7"
format_stamp: "Formatted at 2026-09-25 07:34:13 on dist-test-slave-57fn"
server_key: "0e9503e02373c6be842616fe6f3597bd"
server_key_iv: "7ba649518adf45cc3d3ee26214baeaa9"
server_key_version: "encryptionkey@0"
I20260925 07:34:13.973133 28722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790321392278483-9480-0/minicluster-data/ts-0/wal/instance:
uuid: "27e86fb49a89468788317029f41d4fd7"
format_stamp: "Formatted at 2026-09-25 07:34:13 on dist-test-slave-57fn"
server_key: "0e9503e02373c6be842616fe6f3597bd"
server_key_iv: "7ba649518adf45cc3d3ee26214baeaa9"
server_key_version: "encryptionkey@0"
I20260925 07:34:13.974588 28722 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 07:34:13.975378 28737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:34:13.975558 28722 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 07:34:13.975660 28722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790321392278483-9480-0/minicluster-data/ts-0/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790321392278483-9480-0/minicluster-data/ts-0/wal
uuid: "27e86fb49a89468788317029f41d4fd7"
format_stamp: "Formatted at 2026-09-25 07:34:13 on dist-test-slave-57fn"
server_key: "0e9503e02373c6be842616fe6f3597bd"
server_key_iv: "7ba649518adf45cc3d3ee26214baeaa9"
server_key_version: "encryptionkey@0"
I20260925 07:34:13.975746 28722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790321392278483-9480-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790321392278483-9480-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790321392278483-9480-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:34:13.987188 28722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:34:13.989037 28722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:34:13.989215 28722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:34:13.989446 28722 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:34:13.989773 28722 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:34:13.989831 28722 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:34:13.989866 28722 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:34:13.989900 28722 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:34:13.996387 28722 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.1:33065
I20260925 07:34:13.996430 28850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.1:33065 every 8 connection(s)
I20260925 07:34:13.996780 28722 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790321392278483-9480-0/minicluster-data/ts-0/data/info.pb
Sep 25 07:34:13 dist-test-slave-57fn krb5kdc[28615](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321653, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.1@KRBTEST.COM for kudu/127.9.66.62@KRBTEST.COM
I20260925 07:34:14.005155 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 28722
I20260925 07:34:14.005270 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790321392278483-9480-0/minicluster-data/ts-0/wal/instance
I20260925 07:34:14.005581 9480 external_mini_cluster.cc:1428] Setting key 24bf29ca0959ec94ae0c3cd4451fbd97
I20260925 07:34:14.006294 28700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:13.997552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.1:46251 (local address 127.9.66.62:34589)
0925 07:34:13.997600 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:13.997603 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:13.997778 (+ 175us) server_negotiation.cc:407] Connection header received
0925 07:34:13.998307 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:13.998308 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:13.998320 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:13.998386 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:34:13.999267 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:13.999780 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:14.000782 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:14.000978 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:14.001034 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:14.003121 (+ 2087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:34:14.003122 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:34:14.003123 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:34:14.003130 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:34:14.004572 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:34:14.005026 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:34:14.005028 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:34:14.005028 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:34:14.005101 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:34:14.005489 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:34:14.005491 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:34:14.005491 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:34:14.005665 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:34:14.005758 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:14.006057 (+ 299us) server_negotiation.cc:299] Negotiation successful
0925 07:34:14.006141 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 07:34:14.006343 28853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:13.997472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34589 (local address 127.9.66.1:46251)
0925 07:34:13.997638 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 07:34:13.997679 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 07:34:13.998200 (+ 521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:34:13.998437 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:34:13.998440 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:34:13.998996 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:34:13.999181 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:13.999186 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:13.999867 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:13.999867 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:14.000688 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:14.000691 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:14.001057 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:14.001058 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:14.001152 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:14.001617 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:34:14.001625 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:34:14.003004 (+ 1379us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:34:14.004678 (+ 1674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:34:14.004685 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:34:14.004687 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:34:14.004915 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:34:14.005191 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:34:14.005191 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:34:14.005192 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:34:14.005412 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:34:14.005738 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:34:14.005745 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:34:14.005916 (+ 171us) client_negotiation.cc:769] Sending connection context
0925 07:34:14.006012 (+ 96us) client_negotiation.cc:240] Negotiation successful
0925 07:34:14.006137 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":98,"threads_started":1}
I20260925 07:34:14.006793 28851 heartbeater.cc:347] Connected to a master server at 127.9.66.62:34589
I20260925 07:34:14.006901 28851 heartbeater.cc:464] Registering TS with master...
I20260925 07:34:14.007110 28851 heartbeater.cc:511] Master 127.9.66.62:34589 requested a full tablet report, sending...
I20260925 07:34:14.007687 28660 ts_manager.cc:194] Registered new tserver with Master: 27e86fb49a89468788317029f41d4fd7 (127.9.66.1:33065)
I20260925 07:34:14.008445 28660 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.66.1@KRBTEST.COM'} at 127.9.66.1:46251
WARNING: no policy specified for kudu/127.9.66.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.66.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.66.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.2.keytab.
Entry for principal kudu/127.9.66.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.2.keytab.
I20260925 07:34:14.032161 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790321392278483-9480-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790321392278483-9480-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790321392278483-9480-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790321392278483-9480-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.66.2:0
--local_ip_for_outbound_sockets=127.9.66.2
--webserver_interface=127.9.66.2
--webserver_port=0
--tserver_master_addrs=127.9.66.62:34589
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.2.keytab
--principal=kudu/127.9.66.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:43503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:34:14.111613 28856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:34:14.111827 28856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:34:14.111866 28856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:34:14.113029 28856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:34:14.113098 28856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:34:14.113145 28856 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.66.2
I20260925 07:34:14.114611 28856 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:43503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790321392278483-9480-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790321392278483-9480-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.2.keytab
--principal=kudu/127.9.66.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.66.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790321392278483-9480-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.66.2
--webserver_port=0
--tserver_master_addrs=127.9.66.62:34589
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28856
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.66.2
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790321392278483-9480-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:34:14.114895 28856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:34:14.115113 28856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:34:14.120019 28862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:34:14.120122 28861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:34:14.120015 28864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:34:14.120373 28856 server_base.cc:1034] running on GCE node
I20260925 07:34:14.120546 28856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:34:14.120715 28856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:34:14.121850 28856 hybrid_clock.cc:648] HybridClock initialized: now 1790321654121826 us; error 48 us; skew 500 ppm
Sep 25 07:34:14 dist-test-slave-57fn krb5kdc[28615](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321654, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:34:14.124294 28856 init.cc:377] Logged in from keytab as kudu/127.9.66.2@KRBTEST.COM (short username kudu)
I20260925 07:34:14.124656 28856 webserver.cc:460] Webserver started at http://127.9.66.2:46071/ using document root <none> and password file <none>
I20260925 07:34:14.124826 28856 fs_manager.cc:359] Metadata directory not provided
I20260925 07:34:14.124886 28856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:34:14.124969 28856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:34:14.126003 28856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790321392278483-9480-0/minicluster-data/ts-1/data/instance:
uuid: "3e4746add6f64d2ca7af001cfea59293"
format_stamp: "Formatted at 2026-09-25 07:34:14 on dist-test-slave-57fn"
server_key: "72f25ce55724f7d8ac3f8d309c8a63f4"
server_key_iv: "ccc8190d64a2950413b8201ca37cfd83"
server_key_version: "encryptionkey@0"
I20260925 07:34:14.126353 28856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790321392278483-9480-0/minicluster-data/ts-1/wal/instance:
uuid: "3e4746add6f64d2ca7af001cfea59293"
format_stamp: "Formatted at 2026-09-25 07:34:14 on dist-test-slave-57fn"
server_key: "72f25ce55724f7d8ac3f8d309c8a63f4"
server_key_iv: "ccc8190d64a2950413b8201ca37cfd83"
server_key_version: "encryptionkey@0"
I20260925 07:34:14.127619 28856 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 07:34:14.128317 28871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:34:14.128516 28856 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:34:14.128609 28856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790321392278483-9480-0/minicluster-data/ts-1/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790321392278483-9480-0/minicluster-data/ts-1/wal
uuid: "3e4746add6f64d2ca7af001cfea59293"
format_stamp: "Formatted at 2026-09-25 07:34:14 on dist-test-slave-57fn"
server_key: "72f25ce55724f7d8ac3f8d309c8a63f4"
server_key_iv: "ccc8190d64a2950413b8201ca37cfd83"
server_key_version: "encryptionkey@0"
I20260925 07:34:14.128679 28856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790321392278483-9480-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790321392278483-9480-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790321392278483-9480-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:34:14.143683 28856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:34:14.145473 28856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:34:14.145682 28856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:34:14.145907 28856 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:34:14.146202 28856 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:34:14.146232 28856 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:34:14.146251 28856 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:34:14.146292 28856 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:34:14.151654 28856 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.2:41477
I20260925 07:34:14.151767 28984 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.2:41477 every 8 connection(s)
I20260925 07:34:14.152016 28856 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790321392278483-9480-0/minicluster-data/ts-1/data/info.pb
I20260925 07:34:14.156553 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 28856
I20260925 07:34:14.156698 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790321392278483-9480-0/minicluster-data/ts-1/wal/instance
I20260925 07:34:14.156905 9480 external_mini_cluster.cc:1428] Setting key 58d876cf7d0eddf28615a71ab6a049de
Sep 25 07:34:14 dist-test-slave-57fn krb5kdc[28615](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321654, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.2@KRBTEST.COM for kudu/127.9.66.62@KRBTEST.COM
I20260925 07:34:14.161698 28700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:14.152787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.2:51851 (local address 127.9.66.62:34589)
0925 07:34:14.152870 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:14.152872 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:14.153043 (+ 171us) server_negotiation.cc:407] Connection header received
0925 07:34:14.154031 (+ 988us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:14.154032 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:14.154046 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:14.154095 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:34:14.154728 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:14.155303 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:14.156307 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:14.156529 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:14.156618 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:14.158984 (+ 2366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:34:14.158985 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:34:14.158986 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:34:14.158993 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:34:14.160399 (+ 1406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:34:14.160781 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:34:14.160782 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:34:14.160782 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:34:14.160839 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:34:14.161077 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:34:14.161078 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:34:14.161078 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:34:14.161184 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:34:14.161259 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:14.161530 (+ 271us) server_negotiation.cc:299] Negotiation successful
0925 07:34:14.161615 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 07:34:14.161914 28987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:14.152758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34589 (local address 127.9.66.2:51851)
0925 07:34:14.152912 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 07:34:14.152951 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 07:34:14.153911 (+ 960us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:34:14.154169 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:34:14.154172 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:34:14.154422 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:34:14.154601 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:14.154605 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:14.155428 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:14.155429 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:14.156198 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:14.156201 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:14.156628 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:14.156629 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:14.156765 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:14.157351 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:34:14.157358 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:34:14.158874 (+ 1516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:34:14.160494 (+ 1620us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:34:14.160500 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:34:14.160502 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:34:14.160707 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:34:14.160905 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:34:14.160905 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:34:14.160906 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:34:14.161011 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:34:14.161250 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:34:14.161256 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:34:14.161388 (+ 132us) client_negotiation.cc:769] Sending connection context
0925 07:34:14.161489 (+ 101us) client_negotiation.cc:240] Negotiation successful
0925 07:34:14.161647 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":66,"threads_started":1}
I20260925 07:34:14.162290 28985 heartbeater.cc:347] Connected to a master server at 127.9.66.62:34589
I20260925 07:34:14.162388 28985 heartbeater.cc:464] Registering TS with master...
I20260925 07:34:14.162578 28985 heartbeater.cc:511] Master 127.9.66.62:34589 requested a full tablet report, sending...
I20260925 07:34:14.162895 28660 ts_manager.cc:194] Registered new tserver with Master: 3e4746add6f64d2ca7af001cfea59293 (127.9.66.2:41477)
I20260925 07:34:14.163236 28660 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.66.2@KRBTEST.COM'} at 127.9.66.2:51851
WARNING: no policy specified for kudu/127.9.66.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.66.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.66.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.3.keytab.
Entry for principal kudu/127.9.66.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.3.keytab.
I20260925 07:34:14.184007 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790321392278483-9480-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790321392278483-9480-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790321392278483-9480-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790321392278483-9480-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.66.3:0
--local_ip_for_outbound_sockets=127.9.66.3
--webserver_interface=127.9.66.3
--webserver_port=0
--tserver_master_addrs=127.9.66.62:34589
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.3.keytab
--principal=kudu/127.9.66.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:43503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:34:14.262411 28990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:34:14.262593 28990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:34:14.262629 28990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:34:14.263646 28990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:34:14.263680 28990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:34:14.263705 28990 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.66.3
I20260925 07:34:14.265009 28990 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:43503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790321392278483-9480-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790321392278483-9480-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.3.keytab
--principal=kudu/127.9.66.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.66.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790321392278483-9480-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.66.3
--webserver_port=0
--tserver_master_addrs=127.9.66.62:34589
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28990
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.66.3
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790321392278483-9480-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:34:14.265189 28990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:34:14.265333 28990 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:34:14.270006 28996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:34:14.269995 28998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:34:14.269989 28995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:34:14.270320 28990 server_base.cc:1034] running on GCE node
I20260925 07:34:14.270495 28990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:34:14.270664 28990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:34:14.271793 28990 hybrid_clock.cc:648] HybridClock initialized: now 1790321654271776 us; error 32 us; skew 500 ppm
Sep 25 07:34:14 dist-test-slave-57fn krb5kdc[28615](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321654, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:34:14.274197 28990 init.cc:377] Logged in from keytab as kudu/127.9.66.3@KRBTEST.COM (short username kudu)
I20260925 07:34:14.274576 28990 webserver.cc:460] Webserver started at http://127.9.66.3:38389/ using document root <none> and password file <none>
I20260925 07:34:14.274786 28990 fs_manager.cc:359] Metadata directory not provided
I20260925 07:34:14.274835 28990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:34:14.274974 28990 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:34:14.275908 28990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790321392278483-9480-0/minicluster-data/ts-2/data/instance:
uuid: "d74fc9a01fbf45a3b022271cf1c2140c"
format_stamp: "Formatted at 2026-09-25 07:34:14 on dist-test-slave-57fn"
server_key: "2bc7605ce0f91e47c63d34c3f2a5ac81"
server_key_iv: "8b44d0bcfa5dca6635ab6c766a4e62d6"
server_key_version: "encryptionkey@0"
I20260925 07:34:14.276250 28990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790321392278483-9480-0/minicluster-data/ts-2/wal/instance:
uuid: "d74fc9a01fbf45a3b022271cf1c2140c"
format_stamp: "Formatted at 2026-09-25 07:34:14 on dist-test-slave-57fn"
server_key: "2bc7605ce0f91e47c63d34c3f2a5ac81"
server_key_iv: "8b44d0bcfa5dca6635ab6c766a4e62d6"
server_key_version: "encryptionkey@0"
I20260925 07:34:14.277501 28990 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 07:34:14.278275 29005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:34:14.278481 28990 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 07:34:14.278549 28990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790321392278483-9480-0/minicluster-data/ts-2/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790321392278483-9480-0/minicluster-data/ts-2/wal
uuid: "d74fc9a01fbf45a3b022271cf1c2140c"
format_stamp: "Formatted at 2026-09-25 07:34:14 on dist-test-slave-57fn"
server_key: "2bc7605ce0f91e47c63d34c3f2a5ac81"
server_key_iv: "8b44d0bcfa5dca6635ab6c766a4e62d6"
server_key_version: "encryptionkey@0"
I20260925 07:34:14.278597 28990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790321392278483-9480-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790321392278483-9480-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790321392278483-9480-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:34:14.292212 28990 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:34:14.294354 28990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:34:14.294523 28990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:34:14.294728 28990 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:34:14.295009 28990 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:34:14.295063 28990 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:34:14.295101 28990 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:34:14.295130 28990 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:34:14.300817 28990 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.3:40563
I20260925 07:34:14.300885 29118 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.3:40563 every 8 connection(s)
I20260925 07:34:14.301177 28990 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790321392278483-9480-0/minicluster-data/ts-2/data/info.pb
Sep 25 07:34:14 dist-test-slave-57fn krb5kdc[28615](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321654, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.3@KRBTEST.COM for kudu/127.9.66.62@KRBTEST.COM
I20260925 07:34:14.308337 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 28990
I20260925 07:34:14.308467 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790321392278483-9480-0/minicluster-data/ts-2/wal/instance
I20260925 07:34:14.308655 9480 external_mini_cluster.cc:1428] Setting key 01ed4a76cad3346dec171ee9d88f86ab
I20260925 07:34:14.310164 28700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:14.301867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.3:35183 (local address 127.9.66.62:34589)
0925 07:34:14.301915 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:14.301925 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:14.302082 (+ 157us) server_negotiation.cc:407] Connection header received
0925 07:34:14.302540 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:14.302541 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:14.302550 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:14.302602 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:34:14.303404 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:14.303978 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:14.304974 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:14.305148 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:14.305196 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:14.307212 (+ 2016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:34:14.307214 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:34:14.307214 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:34:14.307221 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:34:14.308642 (+ 1421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:34:14.309129 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:34:14.309130 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:34:14.309131 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:34:14.309207 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:34:14.309497 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:34:14.309498 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:34:14.309498 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:34:14.309653 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:34:14.309723 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:14.310031 (+ 308us) server_negotiation.cc:299] Negotiation successful
0925 07:34:14.310102 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 07:34:14.310221 29121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:14.301837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:34589 (local address 127.9.66.3:35183)
0925 07:34:14.301983 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 07:34:14.302019 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 07:34:14.302441 (+ 422us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:34:14.302648 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:34:14.302651 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:34:14.303143 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:34:14.303319 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:14.303323 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:14.304097 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:14.304099 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:14.304900 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:14.304902 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:14.305217 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:14.305218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:14.305313 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:14.305833 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:34:14.305840 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:34:14.307093 (+ 1253us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:34:14.308731 (+ 1638us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:34:14.308737 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:34:14.308739 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:34:14.309060 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:34:14.309309 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:34:14.309310 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:34:14.309311 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:34:14.309429 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:34:14.309742 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:34:14.309748 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:34:14.309876 (+ 128us) client_negotiation.cc:769] Sending connection context
0925 07:34:14.309971 (+ 95us) client_negotiation.cc:240] Negotiation successful
0925 07:34:14.310080 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":82,"threads_started":1}
I20260925 07:34:14.310635 29119 heartbeater.cc:347] Connected to a master server at 127.9.66.62:34589
I20260925 07:34:14.310719 29119 heartbeater.cc:464] Registering TS with master...
I20260925 07:34:14.310846 29119 heartbeater.cc:511] Master 127.9.66.62:34589 requested a full tablet report, sending...
I20260925 07:34:14.311199 28660 ts_manager.cc:194] Registered new tserver with Master: d74fc9a01fbf45a3b022271cf1c2140c (127.9.66.3:40563)
I20260925 07:34:14.311600 28660 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.66.3@KRBTEST.COM'} at 127.9.66.3:35183
I20260925 07:34:14.320613 9480 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 07:34:14.327571 28700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:14.321884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58756 (local address 127.9.66.62:34589)
0925 07:34:14.321942 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:14.321944 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:14.322058 (+ 114us) server_negotiation.cc:407] Connection header received
0925 07:34:14.322092 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:14.322093 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:14.322101 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:14.322142 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:34:14.322659 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:14.323120 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:14.323800 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:14.323983 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:14.324036 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:14.325128 (+ 1092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:34:14.325129 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:34:14.325129 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:34:14.325135 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:34:14.326361 (+ 1226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:34:14.326768 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:34:14.326769 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:34:14.326769 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:34:14.326821 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:34:14.327024 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:34:14.327025 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:34:14.327025 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:34:14.327160 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:34:14.327276 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:14.327399 (+ 123us) server_negotiation.cc:299] Negotiation successful
0925 07:34:14.327472 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 07:34:14.328969 28660 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58756:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 07:34:14.330582 28660 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 07:34:14.337188 29130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:14.332848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.3:40563 (local address 127.0.0.1:37688)
0925 07:34:14.333644 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0925 07:34:14.333654 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 07:34:14.333699 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:34:14.333961 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:34:14.333962 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:34:14.333964 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:34:14.333997 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:14.333998 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:14.334817 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:14.334818 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:14.336124 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:14.336126 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:14.336878 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:14.336879 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:14.336951 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:14.336965 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 07:34:14.337065 (+ 100us) client_negotiation.cc:240] Negotiation successful
0925 07:34:14.337118 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":784,"mutex_wait_us":1,"thread_start_us":98,"threads_started":1}
I20260925 07:34:14.337199 29134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:14.333056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37688 (local address 127.9.66.3:40563)
0925 07:34:14.333219 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:14.333223 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:14.333761 (+ 538us) server_negotiation.cc:407] Connection header received
0925 07:34:14.333800 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:14.333802 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:14.333816 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:14.333889 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:34:14.334066 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:14.334718 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:14.336230 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:14.336771 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:14.336887 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:14.336947 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:14.337081 (+ 134us) server_negotiation.cc:299] Negotiation successful
0925 07:34:14.337122 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":74,"threads_started":1}
I20260925 07:34:14.337369 29132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:14.332838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.1:33065 (local address 127.0.0.1:56208)
0925 07:34:14.333495 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0925 07:34:14.333501 (+ 6us) client_negotiation.cc:174] Beginning negotiation
0925 07:34:14.333580 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:34:14.333885 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:34:14.333886 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:34:14.333888 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:34:14.333936 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:14.333938 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:14.335202 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:14.335203 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:14.336122 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:14.336124 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:14.337148 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:14.337149 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:14.337214 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:14.337220 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 07:34:14.337268 (+ 48us) client_negotiation.cc:240] Negotiation successful
0925 07:34:14.337303 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":646,"mutex_wait_us":7,"thread_start_us":55,"threads_started":1}
I20260925 07:34:14.337587 29131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:14.332838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.2:41477 (local address 127.0.0.1:43380)
0925 07:34:14.333299 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0925 07:34:14.333333 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 07:34:14.333417 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:34:14.333737 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:34:14.333739 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:34:14.333741 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:34:14.333806 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:14.333808 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:14.335153 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:14.335154 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:14.336246 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:14.336248 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:14.337146 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:14.337147 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:14.337214 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:14.337221 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 07:34:14.337458 (+ 237us) client_negotiation.cc:240] Negotiation successful
0925 07:34:14.337504 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"mutex_wait_us":29,"spinlock_wait_cycles":768,"thread_start_us":111,"threads_started":1}
I20260925 07:34:14.337684 29133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:14.333031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56208 (local address 127.9.66.1:33065)
0925 07:34:14.333191 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:14.333195 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:14.333647 (+ 452us) server_negotiation.cc:407] Connection header received
0925 07:34:14.333708 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:14.333710 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:14.333727 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:14.333797 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:34:14.334019 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:14.335074 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:14.336207 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:14.336980 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:14.337055 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:14.337109 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:14.337547 (+ 438us) server_negotiation.cc:299] Negotiation successful
0925 07:34:14.337593 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":80,"threads_started":1}
I20260925 07:34:14.337823 29135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:14.333122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43380 (local address 127.9.66.2:41477)
0925 07:34:14.333264 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:14.333270 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:14.333375 (+ 105us) server_negotiation.cc:407] Connection header received
0925 07:34:14.333547 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:14.333550 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:14.333567 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:14.333641 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:34:14.333914 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:14.335020 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:14.336318 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:14.337002 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:14.337056 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:14.337109 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:14.337685 (+ 576us) server_negotiation.cc:299] Negotiation successful
0925 07:34:14.337730 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":79,"threads_started":1}
I20260925 07:34:14.338178 29053 tablet_service.cc:1467] Processing CreateTablet for tablet 4ce1a278e2bf44349f4f97df9af4de06 (DEFAULT_TABLE table=test-table [id=89bc954b88904bcdbc52a54b194b5719]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:34:14.338495 29053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ce1a278e2bf44349f4f97df9af4de06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:34:14.338384 28785 tablet_service.cc:1467] Processing CreateTablet for tablet 4ce1a278e2bf44349f4f97df9af4de06 (DEFAULT_TABLE table=test-table [id=89bc954b88904bcdbc52a54b194b5719]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:34:14.338388 28919 tablet_service.cc:1467] Processing CreateTablet for tablet 4ce1a278e2bf44349f4f97df9af4de06 (DEFAULT_TABLE table=test-table [id=89bc954b88904bcdbc52a54b194b5719]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:34:14.338707 28785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ce1a278e2bf44349f4f97df9af4de06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:34:14.338708 28919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ce1a278e2bf44349f4f97df9af4de06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:34:14.340829 29136 tablet_bootstrap.cc:492] T 4ce1a278e2bf44349f4f97df9af4de06 P d74fc9a01fbf45a3b022271cf1c2140c: Bootstrap starting.
I20260925 07:34:14.341460 29136 tablet_bootstrap.cc:654] T 4ce1a278e2bf44349f4f97df9af4de06 P d74fc9a01fbf45a3b022271cf1c2140c: Neither blocks nor log segments found. Creating new log.
I20260925 07:34:14.341809 29137 tablet_bootstrap.cc:492] T 4ce1a278e2bf44349f4f97df9af4de06 P 3e4746add6f64d2ca7af001cfea59293: Bootstrap starting.
I20260925 07:34:14.342007 29138 tablet_bootstrap.cc:492] T 4ce1a278e2bf44349f4f97df9af4de06 P 27e86fb49a89468788317029f41d4fd7: Bootstrap starting.
I20260925 07:34:14.342099 29136 log.cc:824] T 4ce1a278e2bf44349f4f97df9af4de06 P d74fc9a01fbf45a3b022271cf1c2140c: Log is configured to *not* fsync() on all Append() calls
I20260925 07:34:14.342506 29137 tablet_bootstrap.cc:654] T 4ce1a278e2bf44349f4f97df9af4de06 P 3e4746add6f64d2ca7af001cfea59293: Neither blocks nor log segments found. Creating new log.
I20260925 07:34:14.342682 29138 tablet_bootstrap.cc:654] T 4ce1a278e2bf44349f4f97df9af4de06 P 27e86fb49a89468788317029f41d4fd7: Neither blocks nor log segments found. Creating new log.
I20260925 07:34:14.342906 29136 tablet_bootstrap.cc:492] T 4ce1a278e2bf44349f4f97df9af4de06 P d74fc9a01fbf45a3b022271cf1c2140c: No bootstrap required, opened a new log
I20260925 07:34:14.343007 29136 ts_tablet_manager.cc:1397] T 4ce1a278e2bf44349f4f97df9af4de06 P d74fc9a01fbf45a3b022271cf1c2140c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 07:34:14.343083 29137 log.cc:824] T 4ce1a278e2bf44349f4f97df9af4de06 P 3e4746add6f64d2ca7af001cfea59293: Log is configured to *not* fsync() on all Append() calls
I20260925 07:34:14.343252 29138 log.cc:824] T 4ce1a278e2bf44349f4f97df9af4de06 P 27e86fb49a89468788317029f41d4fd7: Log is configured to *not* fsync() on all Append() calls
I20260925 07:34:14.343918 29137 tablet_bootstrap.cc:492] T 4ce1a278e2bf44349f4f97df9af4de06 P 3e4746add6f64d2ca7af001cfea59293: No bootstrap required, opened a new log
I20260925 07:34:14.343986 29138 tablet_bootstrap.cc:492] T 4ce1a278e2bf44349f4f97df9af4de06 P 27e86fb49a89468788317029f41d4fd7: No bootstrap required, opened a new log
I20260925 07:34:14.343995 29137 ts_tablet_manager.cc:1397] T 4ce1a278e2bf44349f4f97df9af4de06 P 3e4746add6f64d2ca7af001cfea59293: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 07:34:14.344051 29138 ts_tablet_manager.cc:1397] T 4ce1a278e2bf44349f4f97df9af4de06 P 27e86fb49a89468788317029f41d4fd7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 07:34:14.345069 29136 raft_consensus.cc:348] T 4ce1a278e2bf44349f4f97df9af4de06 P d74fc9a01fbf45a3b022271cf1c2140c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27e86fb49a89468788317029f41d4fd7" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 33065 } } peers { permanent_uuid: "3e4746add6f64d2ca7af001cfea59293" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 41477 } } peers { permanent_uuid: "d74fc9a01fbf45a3b022271cf1c2140c" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 40563 } }
I20260925 07:34:14.345264 29136 raft_consensus.cc:374] T 4ce1a278e2bf44349f4f97df9af4de06 P d74fc9a01fbf45a3b022271cf1c2140c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:34:14.345301 29136 raft_consensus.cc:729] T 4ce1a278e2bf44349f4f97df9af4de06 P d74fc9a01fbf45a3b022271cf1c2140c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d74fc9a01fbf45a3b022271cf1c2140c, State: Initialized, Role: FOLLOWER
I20260925 07:34:14.345415 29136 consensus_queue.cc:260] T 4ce1a278e2bf44349f4f97df9af4de06 P d74fc9a01fbf45a3b022271cf1c2140c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27e86fb49a89468788317029f41d4fd7" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 33065 } } peers { permanent_uuid: "3e4746add6f64d2ca7af001cfea59293" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 41477 } } peers { permanent_uuid: "d74fc9a01fbf45a3b022271cf1c2140c" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 40563 } }
I20260925 07:34:14.345701 29136 ts_tablet_manager.cc:1428] T 4ce1a278e2bf44349f4f97df9af4de06 P d74fc9a01fbf45a3b022271cf1c2140c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 07:34:14.345726 29138 raft_consensus.cc:348] T 4ce1a278e2bf44349f4f97df9af4de06 P 27e86fb49a89468788317029f41d4fd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27e86fb49a89468788317029f41d4fd7" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 33065 } } peers { permanent_uuid: "3e4746add6f64d2ca7af001cfea59293" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 41477 } } peers { permanent_uuid: "d74fc9a01fbf45a3b022271cf1c2140c" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 40563 } }
I20260925 07:34:14.345923 29138 raft_consensus.cc:374] T 4ce1a278e2bf44349f4f97df9af4de06 P 27e86fb49a89468788317029f41d4fd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:34:14.345971 29138 raft_consensus.cc:729] T 4ce1a278e2bf44349f4f97df9af4de06 P 27e86fb49a89468788317029f41d4fd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27e86fb49a89468788317029f41d4fd7, State: Initialized, Role: FOLLOWER
I20260925 07:34:14.345927 29137 raft_consensus.cc:348] T 4ce1a278e2bf44349f4f97df9af4de06 P 3e4746add6f64d2ca7af001cfea59293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27e86fb49a89468788317029f41d4fd7" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 33065 } } peers { permanent_uuid: "3e4746add6f64d2ca7af001cfea59293" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 41477 } } peers { permanent_uuid: "d74fc9a01fbf45a3b022271cf1c2140c" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 40563 } }
I20260925 07:34:14.346042 29137 raft_consensus.cc:374] T 4ce1a278e2bf44349f4f97df9af4de06 P 3e4746add6f64d2ca7af001cfea59293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:34:14.346096 29137 raft_consensus.cc:729] T 4ce1a278e2bf44349f4f97df9af4de06 P 3e4746add6f64d2ca7af001cfea59293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e4746add6f64d2ca7af001cfea59293, State: Initialized, Role: FOLLOWER
I20260925 07:34:14.346088 29138 consensus_queue.cc:260] T 4ce1a278e2bf44349f4f97df9af4de06 P 27e86fb49a89468788317029f41d4fd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27e86fb49a89468788317029f41d4fd7" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 33065 } } peers { permanent_uuid: "3e4746add6f64d2ca7af001cfea59293" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 41477 } } peers { permanent_uuid: "d74fc9a01fbf45a3b022271cf1c2140c" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 40563 } }
I20260925 07:34:14.346239 29137 consensus_queue.cc:260] T 4ce1a278e2bf44349f4f97df9af4de06 P 3e4746add6f64d2ca7af001cfea59293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27e86fb49a89468788317029f41d4fd7" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 33065 } } peers { permanent_uuid: "3e4746add6f64d2ca7af001cfea59293" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 41477 } } peers { permanent_uuid: "d74fc9a01fbf45a3b022271cf1c2140c" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 40563 } }
I20260925 07:34:14.346375 29138 ts_tablet_manager.cc:1428] T 4ce1a278e2bf44349f4f97df9af4de06 P 27e86fb49a89468788317029f41d4fd7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 07:34:14.346504 28985 heartbeater.cc:503] Master 127.9.66.62:34589 was elected leader, sending a full tablet report...
I20260925 07:34:14.346557 29119 heartbeater.cc:503] Master 127.9.66.62:34589 was elected leader, sending a full tablet report...
I20260925 07:34:14.346506 29137 ts_tablet_manager.cc:1428] T 4ce1a278e2bf44349f4f97df9af4de06 P 3e4746add6f64d2ca7af001cfea59293: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 07:34:14.346756 28851 heartbeater.cc:503] Master 127.9.66.62:34589 was elected leader, sending a full tablet report...
I20260925 07:34:14.379769 29142 raft_consensus.cc:482] T 4ce1a278e2bf44349f4f97df9af4de06 P d74fc9a01fbf45a3b022271cf1c2140c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 07:34:14.379870 29142 raft_consensus.cc:504] T 4ce1a278e2bf44349f4f97df9af4de06 P d74fc9a01fbf45a3b022271cf1c2140c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27e86fb49a89468788317029f41d4fd7" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 33065 } } peers { permanent_uuid: "3e4746add6f64d2ca7af001cfea59293" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 41477 } } peers { permanent_uuid: "d74fc9a01fbf45a3b022271cf1c2140c" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 40563 } }
I20260925 07:34:14.380187 29142 leader_election.cc:288] T 4ce1a278e2bf44349f4f97df9af4de06 P d74fc9a01fbf45a3b022271cf1c2140c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27e86fb49a89468788317029f41d4fd7 (127.9.66.1:33065), 3e4746add6f64d2ca7af001cfea59293 (127.9.66.2:41477)
I20260925 07:34:14.383286 29135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:14.380459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.3:48943 (local address 127.9.66.2:41477)
0925 07:34:14.380519 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:14.380521 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:14.380531 (+ 10us) server_negotiation.cc:407] Connection header received
0925 07:34:14.380558 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:14.380558 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:14.380569 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:14.380623 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:34:14.380766 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:14.381593 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:14.382521 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:14.382925 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:14.382978 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:14.382986 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:14.383174 (+ 188us) server_negotiation.cc:299] Negotiation successful
0925 07:34:14.383215 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 07:34:14.383287 29121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:14.380332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.2:41477 (local address 127.9.66.3:48943)
0925 07:34:14.380387 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0925 07:34:14.380398 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 07:34:14.380447 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:34:14.380638 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:34:14.380639 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:34:14.380641 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:34:14.380713 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:14.380715 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:14.381647 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:14.381648 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:14.382430 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:14.382432 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:14.383011 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:14.383012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:14.383087 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:14.383091 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 07:34:14.383139 (+ 48us) client_negotiation.cc:240] Negotiation successful
0925 07:34:14.383192 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":7}
I20260925 07:34:14.383306 29133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:14.380395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.3:58771 (local address 127.9.66.1:33065)
0925 07:34:14.380471 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:14.380473 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:14.380597 (+ 124us) server_negotiation.cc:407] Connection header received
0925 07:34:14.380640 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:14.380641 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:14.380652 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:14.380724 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:34:14.380933 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:14.381627 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:14.382600 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:14.382939 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:14.383035 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:14.383044 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:14.383207 (+ 163us) server_negotiation.cc:299] Negotiation successful
0925 07:34:14.383247 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 07:34:14.383315 29145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:14.380342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.1:33065 (local address 127.9.66.3:58771)
0925 07:34:14.380523 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0925 07:34:14.380533 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 07:34:14.380594 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:34:14.380726 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:34:14.380727 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:34:14.380728 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:34:14.380842 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:14.380844 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:14.381708 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:14.381709 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:14.382516 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:14.382518 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:14.383059 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:14.383060 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:14.383112 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:14.383115 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 07:34:14.383187 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 07:34:14.383223 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":111,"threads_started":1}
I20260925 07:34:14.383771 28939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ce1a278e2bf44349f4f97df9af4de06" candidate_uuid: "d74fc9a01fbf45a3b022271cf1c2140c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e4746add6f64d2ca7af001cfea59293" is_pre_election: true
I20260925 07:34:14.383920 28939 raft_consensus.cc:2454] T 4ce1a278e2bf44349f4f97df9af4de06 P 3e4746add6f64d2ca7af001cfea59293 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d74fc9a01fbf45a3b022271cf1c2140c in term 0.
I20260925 07:34:14.383998 28805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ce1a278e2bf44349f4f97df9af4de06" candidate_uuid: "d74fc9a01fbf45a3b022271cf1c2140c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27e86fb49a89468788317029f41d4fd7" is_pre_election: true
I20260925 07:34:14.384158 28805 raft_consensus.cc:2454] T 4ce1a278e2bf44349f4f97df9af4de06 P 27e86fb49a89468788317029f41d4fd7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d74fc9a01fbf45a3b022271cf1c2140c in term 0.
I20260925 07:34:14.384123 29007 leader_election.cc:302] T 4ce1a278e2bf44349f4f97df9af4de06 P d74fc9a01fbf45a3b022271cf1c2140c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e4746add6f64d2ca7af001cfea59293, d74fc9a01fbf45a3b022271cf1c2140c; no voters:
I20260925 07:34:14.384279 29142 raft_consensus.cc:2781] T 4ce1a278e2bf44349f4f97df9af4de06 P d74fc9a01fbf45a3b022271cf1c2140c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 07:34:14.384347 29142 raft_consensus.cc:482] T 4ce1a278e2bf44349f4f97df9af4de06 P d74fc9a01fbf45a3b022271cf1c2140c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 07:34:14.384373 29142 raft_consensus.cc:3037] T 4ce1a278e2bf44349f4f97df9af4de06 P d74fc9a01fbf45a3b022271cf1c2140c [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:34:14.385025 29142 raft_consensus.cc:504] T 4ce1a278e2bf44349f4f97df9af4de06 P d74fc9a01fbf45a3b022271cf1c2140c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27e86fb49a89468788317029f41d4fd7" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 33065 } } peers { permanent_uuid: "3e4746add6f64d2ca7af001cfea59293" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 41477 } } peers { permanent_uuid: "d74fc9a01fbf45a3b022271cf1c2140c" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 40563 } }
I20260925 07:34:14.385285 29142 leader_election.cc:288] T 4ce1a278e2bf44349f4f97df9af4de06 P d74fc9a01fbf45a3b022271cf1c2140c [CANDIDATE]: Term 1 election: Requested vote from peers 27e86fb49a89468788317029f41d4fd7 (127.9.66.1:33065), 3e4746add6f64d2ca7af001cfea59293 (127.9.66.2:41477)
I20260925 07:34:14.385298 28939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ce1a278e2bf44349f4f97df9af4de06" candidate_uuid: "d74fc9a01fbf45a3b022271cf1c2140c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e4746add6f64d2ca7af001cfea59293"
I20260925 07:34:14.385299 28805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ce1a278e2bf44349f4f97df9af4de06" candidate_uuid: "d74fc9a01fbf45a3b022271cf1c2140c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27e86fb49a89468788317029f41d4fd7"
I20260925 07:34:14.385361 28939 raft_consensus.cc:3037] T 4ce1a278e2bf44349f4f97df9af4de06 P 3e4746add6f64d2ca7af001cfea59293 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:34:14.385396 28805 raft_consensus.cc:3037] T 4ce1a278e2bf44349f4f97df9af4de06 P 27e86fb49a89468788317029f41d4fd7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:34:14.386163 28805 raft_consensus.cc:2454] T 4ce1a278e2bf44349f4f97df9af4de06 P 27e86fb49a89468788317029f41d4fd7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d74fc9a01fbf45a3b022271cf1c2140c in term 1.
I20260925 07:34:14.386173 28939 raft_consensus.cc:2454] T 4ce1a278e2bf44349f4f97df9af4de06 P 3e4746add6f64d2ca7af001cfea59293 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d74fc9a01fbf45a3b022271cf1c2140c in term 1.
I20260925 07:34:14.386341 29009 leader_election.cc:302] T 4ce1a278e2bf44349f4f97df9af4de06 P d74fc9a01fbf45a3b022271cf1c2140c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27e86fb49a89468788317029f41d4fd7, d74fc9a01fbf45a3b022271cf1c2140c; no voters:
I20260925 07:34:14.386444 29142 raft_consensus.cc:2781] T 4ce1a278e2bf44349f4f97df9af4de06 P d74fc9a01fbf45a3b022271cf1c2140c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:34:14.386621 29142 raft_consensus.cc:686] T 4ce1a278e2bf44349f4f97df9af4de06 P d74fc9a01fbf45a3b022271cf1c2140c [term 1 LEADER]: Becoming Leader. State: Replica: d74fc9a01fbf45a3b022271cf1c2140c, State: Running, Role: LEADER
I20260925 07:34:14.386752 29142 consensus_queue.cc:237] T 4ce1a278e2bf44349f4f97df9af4de06 P d74fc9a01fbf45a3b022271cf1c2140c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27e86fb49a89468788317029f41d4fd7" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 33065 } } peers { permanent_uuid: "3e4746add6f64d2ca7af001cfea59293" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 41477 } } peers { permanent_uuid: "d74fc9a01fbf45a3b022271cf1c2140c" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 40563 } }
I20260925 07:34:14.387559 28660 catalog_manager.cc:5495] T 4ce1a278e2bf44349f4f97df9af4de06 P d74fc9a01fbf45a3b022271cf1c2140c reported cstate change: term changed from 0 to 1, leader changed from <none> to d74fc9a01fbf45a3b022271cf1c2140c (127.9.66.3). New cstate: current_term: 1 leader_uuid: "d74fc9a01fbf45a3b022271cf1c2140c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27e86fb49a89468788317029f41d4fd7" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 33065 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e4746add6f64d2ca7af001cfea59293" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 41477 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d74fc9a01fbf45a3b022271cf1c2140c" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 40563 } health_report { overall_health: HEALTHY } } }
I20260925 07:34:14.397136 29134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:14.395002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37698 (local address 127.9.66.3:40563)
0925 07:34:14.395057 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:14.395059 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:14.395075 (+ 16us) server_negotiation.cc:407] Connection header received
0925 07:34:14.395105 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:14.395105 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:14.395119 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:14.395167 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 07:34:14.395308 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:14.395764 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:14.396495 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:14.396664 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:14.396710 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:14.396838 (+ 128us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 07:34:14.396930 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 07:34:14.396987 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:14.397017 (+ 30us) server_negotiation.cc:299] Negotiation successful
0925 07:34:14.397032 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 07:34:14.398726 28805 raft_consensus.cc:1261] T 4ce1a278e2bf44349f4f97df9af4de06 P 27e86fb49a89468788317029f41d4fd7 [term 1 FOLLOWER]: Refusing update from remote peer d74fc9a01fbf45a3b022271cf1c2140c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:34:14.398736 28939 raft_consensus.cc:1261] T 4ce1a278e2bf44349f4f97df9af4de06 P 3e4746add6f64d2ca7af001cfea59293 [term 1 FOLLOWER]: Refusing update from remote peer d74fc9a01fbf45a3b022271cf1c2140c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:34:14.399066 29146 consensus_queue.cc:1035] T 4ce1a278e2bf44349f4f97df9af4de06 P d74fc9a01fbf45a3b022271cf1c2140c [LEADER]: Connected to new peer: Peer: permanent_uuid: "27e86fb49a89468788317029f41d4fd7" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 33065 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:34:14.399178 29142 consensus_queue.cc:1035] T 4ce1a278e2bf44349f4f97df9af4de06 P d74fc9a01fbf45a3b022271cf1c2140c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e4746add6f64d2ca7af001cfea59293" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 41477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:34:14.401566 29149 mvcc.cc:205] Tried to move back new op lower bound from 7333157496415076352 to 7333157496368807936. Current Snapshot: MvccSnapshot[applied={T|T < 7333157496415076352}]
I20260925 07:34:14.401674 29150 mvcc.cc:205] Tried to move back new op lower bound from 7333157496415076352 to 7333157496368807936. Current Snapshot: MvccSnapshot[applied={T|T < 7333157496415076352}]
W20260925 07:34:14.402524 28986 tablet.cc:2369] T 4ce1a278e2bf44349f4f97df9af4de06 P 3e4746add6f64d2ca7af001cfea59293: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 07:34:14.403098 29152 mvcc.cc:205] Tried to move back new op lower bound from 7333157496415076352 to 7333157496368807936. Current Snapshot: MvccSnapshot[applied={T|T < 7333157496415076352}]
I20260925 07:34:14.403919 28660 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58756:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 07:34:14.404003 28660 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58756:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 07:34:14.404733 28660 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 97b8026bd9544e9ba5e5ef1e6545169b: Sending DeleteTablet for 3 replicas of tablet 4ce1a278e2bf44349f4f97df9af4de06
I20260925 07:34:14.405066 28919 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ce1a278e2bf44349f4f97df9af4de06 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:34:14 UTC) from {username='kudu', principal='kudu/127.9.66.62@KRBTEST.COM'} at 127.0.0.1:43380
I20260925 07:34:14.405068 28785 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ce1a278e2bf44349f4f97df9af4de06 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:34:14 UTC) from {username='kudu', principal='kudu/127.9.66.62@KRBTEST.COM'} at 127.0.0.1:56208
I20260925 07:34:14.405279 29160 tablet_replica.cc:330] T 4ce1a278e2bf44349f4f97df9af4de06 P 27e86fb49a89468788317029f41d4fd7: stopping tablet replica
I20260925 07:34:14.405272 29053 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ce1a278e2bf44349f4f97df9af4de06 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:34:14 UTC) from {username='kudu', principal='kudu/127.9.66.62@KRBTEST.COM'} at 127.0.0.1:37688
I20260925 07:34:14.405359 29160 raft_consensus.cc:2229] T 4ce1a278e2bf44349f4f97df9af4de06 P 27e86fb49a89468788317029f41d4fd7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 07:34:14.405455 29160 raft_consensus.cc:2258] T 4ce1a278e2bf44349f4f97df9af4de06 P 27e86fb49a89468788317029f41d4fd7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:34:14.405462 29161 tablet_replica.cc:330] T 4ce1a278e2bf44349f4f97df9af4de06 P d74fc9a01fbf45a3b022271cf1c2140c: stopping tablet replica
I20260925 07:34:14.405573 29161 raft_consensus.cc:2229] T 4ce1a278e2bf44349f4f97df9af4de06 P d74fc9a01fbf45a3b022271cf1c2140c [term 1 LEADER]: Raft consensus shutting down.
I20260925 07:34:14.405663 29161 raft_consensus.cc:2258] T 4ce1a278e2bf44349f4f97df9af4de06 P d74fc9a01fbf45a3b022271cf1c2140c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:34:14.405905 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 28722
I20260925 07:34:14.406008 29161 ts_tablet_manager.cc:1905] T 4ce1a278e2bf44349f4f97df9af4de06 P d74fc9a01fbf45a3b022271cf1c2140c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 07:34:14.406008 29160 ts_tablet_manager.cc:1905] T 4ce1a278e2bf44349f4f97df9af4de06 P 27e86fb49a89468788317029f41d4fd7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 07:34:14.406736 29159 tablet_replica.cc:330] T 4ce1a278e2bf44349f4f97df9af4de06 P 3e4746add6f64d2ca7af001cfea59293: stopping tablet replica
I20260925 07:34:14.406829 29159 raft_consensus.cc:2229] T 4ce1a278e2bf44349f4f97df9af4de06 P 3e4746add6f64d2ca7af001cfea59293 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 07:34:14.406914 29159 raft_consensus.cc:2258] T 4ce1a278e2bf44349f4f97df9af4de06 P 3e4746add6f64d2ca7af001cfea59293 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:34:14.407353 29159 ts_tablet_manager.cc:1905] T 4ce1a278e2bf44349f4f97df9af4de06 P 3e4746add6f64d2ca7af001cfea59293: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 07:34:14.407743 29161 ts_tablet_manager.cc:1918] T 4ce1a278e2bf44349f4f97df9af4de06 P d74fc9a01fbf45a3b022271cf1c2140c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 07:34:14.407804 29161 log.cc:1196] T 4ce1a278e2bf44349f4f97df9af4de06 P d74fc9a01fbf45a3b022271cf1c2140c: Deleting WAL directory at /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790321392278483-9480-0/minicluster-data/ts-2/wal/wals/4ce1a278e2bf44349f4f97df9af4de06
I20260925 07:34:14.408077 29161 ts_tablet_manager.cc:1939] T 4ce1a278e2bf44349f4f97df9af4de06 P d74fc9a01fbf45a3b022271cf1c2140c: Deleting consensus metadata
I20260925 07:34:14.408601 28644 catalog_manager.cc:4841] TS d74fc9a01fbf45a3b022271cf1c2140c (127.9.66.3:40563): tablet 4ce1a278e2bf44349f4f97df9af4de06 (table test-table [id=89bc954b88904bcdbc52a54b194b5719]) successfully deleted
I20260925 07:34:14.409247 29159 ts_tablet_manager.cc:1918] T 4ce1a278e2bf44349f4f97df9af4de06 P 3e4746add6f64d2ca7af001cfea59293: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 07:34:14.409292 29159 log.cc:1196] T 4ce1a278e2bf44349f4f97df9af4de06 P 3e4746add6f64d2ca7af001cfea59293: Deleting WAL directory at /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790321392278483-9480-0/minicluster-data/ts-1/wal/wals/4ce1a278e2bf44349f4f97df9af4de06
I20260925 07:34:14.409472 29159 ts_tablet_manager.cc:1939] T 4ce1a278e2bf44349f4f97df9af4de06 P 3e4746add6f64d2ca7af001cfea59293: Deleting consensus metadata
I20260925 07:34:14.409960 28645 catalog_manager.cc:4841] TS 3e4746add6f64d2ca7af001cfea59293 (127.9.66.2:41477): tablet 4ce1a278e2bf44349f4f97df9af4de06 (table test-table [id=89bc954b88904bcdbc52a54b194b5719]) successfully deleted
W20260925 07:34:14.413849 28647 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.66.1:33065 (error 108)
I20260925 07:34:14.414096 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 28856
I20260925 07:34:14.414752 29132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:14.414220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.1:33065 (local address 127.0.0.1:56216)
0925 07:34:14.414687 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0925 07:34:14.414721 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.1:33065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":447}
W20260925 07:34:14.415733 28647 catalog_manager.cc:4584] TS 27e86fb49a89468788317029f41d4fd7 (127.9.66.1:33065): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4ce1a278e2bf44349f4f97df9af4de06: Network error: Client connection negotiation failed: client connection to 127.9.66.1:33065: connect: Connection refused (error 111)
I20260925 07:34:14.420387 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 28990
I20260925 07:34:14.424889 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 28628
2026-09-25T07:34:14Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (1870 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-25T07:34:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T07:34:14Z Disabled control of system clock
I20260925 07:34:14.451565 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790321392278483-9480-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790321392278483-9480-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790321392278483-9480-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.66.62:40053
--webserver_interface=127.9.66.62
--webserver_port=0
--builtin_ntp_servers=127.9.66.20:35003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.66.62:40053
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 07:34:14.533715 29166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:34:14.533910 29166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:34:14.533928 29166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:34:14.535090 29166 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:34:14.535125 29166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:34:14.535137 29166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:34:14.535149 29166 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:34:14.535194 29166 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:34:14.536429 29166 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:35003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790321392278483-9480-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790321392278483-9480-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.66.62:40053
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.66.62:40053
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.66.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29166
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790321392278483-9480-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:34:14.536656 29166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:34:14.536938 29166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:34:14.541491 29172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:34:14.541497 29171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:34:14.541553 29174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:34:14.541702 29166 server_base.cc:1034] running on GCE node
I20260925 07:34:14.541980 29166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:34:14.542299 29166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:34:14.543459 29166 hybrid_clock.cc:648] HybridClock initialized: now 1790321654543455 us; error 38 us; skew 500 ppm
I20260925 07:34:14.544695 29166 webserver.cc:460] Webserver started at http://127.9.66.62:32915/ using document root <none> and password file <none>
I20260925 07:34:14.544938 29166 fs_manager.cc:359] Metadata directory not provided
I20260925 07:34:14.545009 29166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:34:14.545152 29166 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:34:14.546234 29166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790321392278483-9480-0/minicluster-data/master-0/data/instance:
uuid: "46c2640c72a94770b5333ac0a44b551e"
format_stamp: "Formatted at 2026-09-25 07:34:14 on dist-test-slave-57fn"
server_key: "0c56c779be463278961d8c6db9262c65"
server_key_iv: "698ad6d48f439d1730359bdc12881587"
server_key_version: "encryptionkey@0"
I20260925 07:34:14.546600 29166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790321392278483-9480-0/minicluster-data/master-0/wal/instance:
uuid: "46c2640c72a94770b5333ac0a44b551e"
format_stamp: "Formatted at 2026-09-25 07:34:14 on dist-test-slave-57fn"
server_key: "0c56c779be463278961d8c6db9262c65"
server_key_iv: "698ad6d48f439d1730359bdc12881587"
server_key_version: "encryptionkey@0"
I20260925 07:34:14.547888 29166 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 07:34:14.548667 29180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:34:14.548840 29166 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:34:14.548940 29166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790321392278483-9480-0/minicluster-data/master-0/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790321392278483-9480-0/minicluster-data/master-0/wal
uuid: "46c2640c72a94770b5333ac0a44b551e"
format_stamp: "Formatted at 2026-09-25 07:34:14 on dist-test-slave-57fn"
server_key: "0c56c779be463278961d8c6db9262c65"
server_key_iv: "698ad6d48f439d1730359bdc12881587"
server_key_version: "encryptionkey@0"
I20260925 07:34:14.549031 29166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790321392278483-9480-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790321392278483-9480-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790321392278483-9480-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:34:14.558383 29166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:34:14.558671 29166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:34:14.558816 29166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:34:14.562605 29166 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.62:40053
I20260925 07:34:14.562642 29232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.62:40053 every 8 connection(s)
I20260925 07:34:14.562966 29166 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
I20260925 07:34:14.563944 29233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:34:14.565579 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 29166
I20260925 07:34:14.565759 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790321392278483-9480-0/minicluster-data/master-0/wal/instance
I20260925 07:34:14.565973 9480 external_mini_cluster.cc:1428] Setting key 267ced53946c1852bc37a647930c064f
I20260925 07:34:14.566576 29233 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46c2640c72a94770b5333ac0a44b551e: Bootstrap starting.
I20260925 07:34:14.567183 29233 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 46c2640c72a94770b5333ac0a44b551e: Neither blocks nor log segments found. Creating new log.
I20260925 07:34:14.567579 29233 log.cc:824] T 00000000000000000000000000000000 P 46c2640c72a94770b5333ac0a44b551e: Log is configured to *not* fsync() on all Append() calls
I20260925 07:34:14.568365 29233 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46c2640c72a94770b5333ac0a44b551e: No bootstrap required, opened a new log
I20260925 07:34:14.570597 29233 raft_consensus.cc:348] T 00000000000000000000000000000000 P 46c2640c72a94770b5333ac0a44b551e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46c2640c72a94770b5333ac0a44b551e" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 40053 } }
I20260925 07:34:14.570727 29233 raft_consensus.cc:374] T 00000000000000000000000000000000 P 46c2640c72a94770b5333ac0a44b551e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:34:14.570749 29233 raft_consensus.cc:729] T 00000000000000000000000000000000 P 46c2640c72a94770b5333ac0a44b551e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46c2640c72a94770b5333ac0a44b551e, State: Initialized, Role: FOLLOWER
I20260925 07:34:14.570881 29233 consensus_queue.cc:260] T 00000000000000000000000000000000 P 46c2640c72a94770b5333ac0a44b551e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46c2640c72a94770b5333ac0a44b551e" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 40053 } }
I20260925 07:34:14.570991 29233 raft_consensus.cc:388] T 00000000000000000000000000000000 P 46c2640c72a94770b5333ac0a44b551e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 07:34:14.571024 29233 raft_consensus.cc:482] T 00000000000000000000000000000000 P 46c2640c72a94770b5333ac0a44b551e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 07:34:14.571072 29233 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 46c2640c72a94770b5333ac0a44b551e [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:34:14.571766 29233 raft_consensus.cc:504] T 00000000000000000000000000000000 P 46c2640c72a94770b5333ac0a44b551e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46c2640c72a94770b5333ac0a44b551e" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 40053 } }
I20260925 07:34:14.571856 29233 leader_election.cc:302] T 00000000000000000000000000000000 P 46c2640c72a94770b5333ac0a44b551e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 46c2640c72a94770b5333ac0a44b551e; no voters:
I20260925 07:34:14.572049 29233 leader_election.cc:288] T 00000000000000000000000000000000 P 46c2640c72a94770b5333ac0a44b551e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 07:34:14.572082 29238 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 46c2640c72a94770b5333ac0a44b551e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:34:14.572190 29238 raft_consensus.cc:686] T 00000000000000000000000000000000 P 46c2640c72a94770b5333ac0a44b551e [term 1 LEADER]: Becoming Leader. State: Replica: 46c2640c72a94770b5333ac0a44b551e, State: Running, Role: LEADER
I20260925 07:34:14.572336 29233 sys_catalog.cc:564] T 00000000000000000000000000000000 P 46c2640c72a94770b5333ac0a44b551e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:34:14.572433 29238 consensus_queue.cc:237] T 00000000000000000000000000000000 P 46c2640c72a94770b5333ac0a44b551e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46c2640c72a94770b5333ac0a44b551e" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 40053 } }
I20260925 07:34:14.572611 29236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:14.566582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36018 (local address 127.9.66.62:40053)
0925 07:34:14.566788 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:14.566795 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:14.566818 (+ 23us) server_negotiation.cc:407] Connection header received
0925 07:34:14.567282 (+ 464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:14.567290 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:14.567365 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:14.567763 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:34:14.568056 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:14.569148 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:14.570399 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:14.570868 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:14.571004 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:14.571711 (+ 707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:34:14.571715 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:34:14.571715 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 07:34:14.571929 (+ 214us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:34:14.571977 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 07:34:14.571989 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:34:14.572068 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:14.572259 (+ 191us) server_negotiation.cc:299] Negotiation successful
0925 07:34:14.572333 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":84,"threads_started":1}
I20260925 07:34:14.572799 29238 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46c2640c72a94770b5333ac0a44b551e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 46c2640c72a94770b5333ac0a44b551e. Latest consensus state: current_term: 1 leader_uuid: "46c2640c72a94770b5333ac0a44b551e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46c2640c72a94770b5333ac0a44b551e" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 40053 } } }
I20260925 07:34:14.572882 29238 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46c2640c72a94770b5333ac0a44b551e [sys.catalog]: This master's current role is: LEADER
I20260925 07:34:14.573190 29247 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 07:34:14.573738 29247 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 07:34:14.573729 29240 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46c2640c72a94770b5333ac0a44b551e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "46c2640c72a94770b5333ac0a44b551e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46c2640c72a94770b5333ac0a44b551e" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 40053 } } }
I20260925 07:34:14.573956 29240 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46c2640c72a94770b5333ac0a44b551e [sys.catalog]: This master's current role is: LEADER
I20260925 07:34:14.575397 29247 catalog_manager.cc:1347] Generated new cluster ID: 724586013a9d48e38fe295433b19b12d
I20260925 07:34:14.575464 29247 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 07:34:14.593817 29247 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 07:34:14.594511 29247 catalog_manager.cc:1504] Loading token signing keys...
I20260925 07:34:14.597000 29247 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 46c2640c72a94770b5333ac0a44b551e: Generated new TSK 0
I20260925 07:34:14.597162 29247 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 07:34:14.599421 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790321392278483-9480-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790321392278483-9480-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790321392278483-9480-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790321392278483-9480-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.66.1:0
--local_ip_for_outbound_sockets=127.9.66.1
--webserver_interface=127.9.66.1
--webserver_port=0
--tserver_master_addrs=127.9.66.62:40053
--builtin_ntp_servers=127.9.66.20:35003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 07:34:14.679662 29257 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:34:14.679875 29257 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:34:14.679910 29257 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:34:14.681034 29257 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:34:14.681093 29257 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:34:14.681135 29257 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.66.1
I20260925 07:34:14.682518 29257 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:35003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790321392278483-9480-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790321392278483-9480-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.66.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790321392278483-9480-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.66.1
--webserver_port=0
--tserver_master_addrs=127.9.66.62:40053
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29257
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.66.1
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790321392278483-9480-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:34:14.682757 29257 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:34:14.683030 29257 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:34:14.688638 29263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:34:14.688862 29257 server_base.cc:1034] running on GCE node
W20260925 07:34:14.688906 29265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:34:14.688968 29262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:34:14.689217 29257 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:34:14.689471 29257 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:34:14.690642 29257 hybrid_clock.cc:648] HybridClock initialized: now 1790321654690631 us; error 28 us; skew 500 ppm
I20260925 07:34:14.691820 29257 webserver.cc:460] Webserver started at http://127.9.66.1:37189/ using document root <none> and password file <none>
I20260925 07:34:14.692056 29257 fs_manager.cc:359] Metadata directory not provided
I20260925 07:34:14.692123 29257 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:34:14.692250 29257 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:34:14.693266 29257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790321392278483-9480-0/minicluster-data/ts-0/data/instance:
uuid: "890ac1f3a91c469fa2a89a046a7d3786"
format_stamp: "Formatted at 2026-09-25 07:34:14 on dist-test-slave-57fn"
server_key: "25597e79b4c65b3dd77d36bbc07b52ed"
server_key_iv: "b9ef8de7c796185229ff4ef58d59274b"
server_key_version: "encryptionkey@0"
I20260925 07:34:14.693655 29257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790321392278483-9480-0/minicluster-data/ts-0/wal/instance:
uuid: "890ac1f3a91c469fa2a89a046a7d3786"
format_stamp: "Formatted at 2026-09-25 07:34:14 on dist-test-slave-57fn"
server_key: "25597e79b4c65b3dd77d36bbc07b52ed"
server_key_iv: "b9ef8de7c796185229ff4ef58d59274b"
server_key_version: "encryptionkey@0"
I20260925 07:34:14.694891 29257 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:34:14.695626 29271 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:34:14.695812 29257 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 07:34:14.695905 29257 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790321392278483-9480-0/minicluster-data/ts-0/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790321392278483-9480-0/minicluster-data/ts-0/wal
uuid: "890ac1f3a91c469fa2a89a046a7d3786"
format_stamp: "Formatted at 2026-09-25 07:34:14 on dist-test-slave-57fn"
server_key: "25597e79b4c65b3dd77d36bbc07b52ed"
server_key_iv: "b9ef8de7c796185229ff4ef58d59274b"
server_key_version: "encryptionkey@0"
I20260925 07:34:14.696004 29257 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790321392278483-9480-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790321392278483-9480-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790321392278483-9480-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:34:14.737097 29257 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:34:14.737370 29257 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:34:14.737470 29257 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:34:14.737747 29257 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:34:14.738155 29257 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:34:14.738204 29257 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:34:14.738246 29257 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:34:14.738272 29257 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:34:14.743753 29257 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.1:41079
I20260925 07:34:14.743839 29384 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.1:41079 every 8 connection(s)
I20260925 07:34:14.744133 29257 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790321392278483-9480-0/minicluster-data/ts-0/data/info.pb
I20260925 07:34:14.744376 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 29257
I20260925 07:34:14.744525 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790321392278483-9480-0/minicluster-data/ts-0/wal/instance
I20260925 07:34:14.744748 9480 external_mini_cluster.cc:1428] Setting key 0f7354539eec7117fd571c91ea5178c7
I20260925 07:34:14.745823 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790321392278483-9480-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790321392278483-9480-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790321392278483-9480-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790321392278483-9480-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.66.2:0
--local_ip_for_outbound_sockets=127.9.66.2
--webserver_interface=127.9.66.2
--webserver_port=0
--tserver_master_addrs=127.9.66.62:40053
--builtin_ntp_servers=127.9.66.20:35003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 07:34:14.750352 29236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:14.745018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.1:50963 (local address 127.9.66.62:40053)
0925 07:34:14.745059 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:14.745061 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:14.745335 (+ 274us) server_negotiation.cc:407] Connection header received
0925 07:34:14.746102 (+ 767us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:14.746103 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:14.746113 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:14.746149 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:34:14.746994 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:14.747484 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:14.748769 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:14.749117 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:14.749187 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:14.750061 (+ 874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:34:14.750063 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:34:14.750063 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 07:34:14.750069 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:34:14.750086 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 07:34:14.750091 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:34:14.750151 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:14.750207 (+ 56us) server_negotiation.cc:299] Negotiation successful
0925 07:34:14.750235 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 07:34:14.750505 29387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:14.745034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:40053 (local address 127.9.66.1:50963)
0925 07:34:14.745231 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0925 07:34:14.745263 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0925 07:34:14.746002 (+ 739us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:34:14.746222 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:34:14.746227 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:34:14.746236 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:34:14.746874 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:14.746879 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:14.747568 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:14.747569 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:14.748688 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:14.748690 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:14.749195 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:14.749196 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:14.749296 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:14.749926 (+ 630us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 07:34:14.749927 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:34:14.749955 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 07:34:14.749976 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:34:14.750152 (+ 176us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:34:14.750156 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:34:14.750158 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 07:34:14.750206 (+ 48us) client_negotiation.cc:240] Negotiation successful
0925 07:34:14.750278 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":81,"threads_started":1}
I20260925 07:34:14.750995 29385 heartbeater.cc:347] Connected to a master server at 127.9.66.62:40053
I20260925 07:34:14.751086 29385 heartbeater.cc:464] Registering TS with master...
I20260925 07:34:14.751272 29385 heartbeater.cc:511] Master 127.9.66.62:40053 requested a full tablet report, sending...
I20260925 07:34:14.751763 29197 ts_manager.cc:194] Registered new tserver with Master: 890ac1f3a91c469fa2a89a046a7d3786 (127.9.66.1:41079)
I20260925 07:34:14.752360 29197 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.66.1:50963
W20260925 07:34:14.824846 29388 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:34:14.825014 29388 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:34:14.825034 29388 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:34:14.826201 29388 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:34:14.826238 29388 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:34:14.826263 29388 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.66.2
I20260925 07:34:14.827582 29388 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:35003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790321392278483-9480-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790321392278483-9480-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.66.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790321392278483-9480-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.66.2
--webserver_port=0
--tserver_master_addrs=127.9.66.62:40053
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29388
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.66.2
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790321392278483-9480-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:34:14.827802 29388 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:34:14.828099 29388 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:34:14.833112 29393 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:34:14.833122 29396 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:34:14.833215 29388 server_base.cc:1034] running on GCE node
W20260925 07:34:14.833122 29394 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:34:14.833611 29388 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:34:14.833796 29388 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:34:14.834985 29388 hybrid_clock.cc:648] HybridClock initialized: now 1790321654834974 us; error 43 us; skew 500 ppm
I20260925 07:34:14.836040 29388 webserver.cc:460] Webserver started at http://127.9.66.2:40649/ using document root <none> and password file <none>
I20260925 07:34:14.836251 29388 fs_manager.cc:359] Metadata directory not provided
I20260925 07:34:14.836288 29388 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:34:14.836374 29388 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:34:14.837348 29388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790321392278483-9480-0/minicluster-data/ts-1/data/instance:
uuid: "ef1f64d47d80417c9fffa9df47efa71a"
format_stamp: "Formatted at 2026-09-25 07:34:14 on dist-test-slave-57fn"
server_key: "2d2c16a107f2f52073d4d5214cd903bd"
server_key_iv: "4d9adc7196e9ac1adb7ec4412bcac524"
server_key_version: "encryptionkey@0"
I20260925 07:34:14.837698 29388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790321392278483-9480-0/minicluster-data/ts-1/wal/instance:
uuid: "ef1f64d47d80417c9fffa9df47efa71a"
format_stamp: "Formatted at 2026-09-25 07:34:14 on dist-test-slave-57fn"
server_key: "2d2c16a107f2f52073d4d5214cd903bd"
server_key_iv: "4d9adc7196e9ac1adb7ec4412bcac524"
server_key_version: "encryptionkey@0"
I20260925 07:34:14.838850 29388 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 07:34:14.839533 29402 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:34:14.839700 29388 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 07:34:14.839771 29388 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790321392278483-9480-0/minicluster-data/ts-1/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790321392278483-9480-0/minicluster-data/ts-1/wal
uuid: "ef1f64d47d80417c9fffa9df47efa71a"
format_stamp: "Formatted at 2026-09-25 07:34:14 on dist-test-slave-57fn"
server_key: "2d2c16a107f2f52073d4d5214cd903bd"
server_key_iv: "4d9adc7196e9ac1adb7ec4412bcac524"
server_key_version: "encryptionkey@0"
I20260925 07:34:14.839819 29388 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790321392278483-9480-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790321392278483-9480-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790321392278483-9480-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:34:14.850414 29388 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:34:14.850611 29388 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:34:14.850698 29388 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:34:14.850894 29388 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:34:14.851176 29388 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:34:14.851207 29388 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:34:14.851224 29388 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:34:14.851269 29388 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:34:14.856706 29388 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.2:46377
I20260925 07:34:14.856781 29515 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.2:46377 every 8 connection(s)
I20260925 07:34:14.857081 29388 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790321392278483-9480-0/minicluster-data/ts-1/data/info.pb
I20260925 07:34:14.861101 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 29388
I20260925 07:34:14.861217 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790321392278483-9480-0/minicluster-data/ts-1/wal/instance
I20260925 07:34:14.861397 9480 external_mini_cluster.cc:1428] Setting key 07063c8b2dd8df0a59feff0b66f32997
I20260925 07:34:14.861852 29236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:14.857786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.2:46833 (local address 127.9.66.62:40053)
0925 07:34:14.857828 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:14.857830 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:14.858033 (+ 203us) server_negotiation.cc:407] Connection header received
0925 07:34:14.858563 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:14.858564 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:14.858574 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:14.858623 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:34:14.859192 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:14.859679 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:14.860508 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:14.860667 (+ 159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:14.860710 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:14.861577 (+ 867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:34:14.861579 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:34:14.861579 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 07:34:14.861584 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:34:14.861599 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 07:34:14.861603 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:34:14.861649 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:14.861759 (+ 110us) server_negotiation.cc:299] Negotiation successful
0925 07:34:14.861783 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 07:34:14.862142 29518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:14.857768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:40053 (local address 127.9.66.2:46833)
0925 07:34:14.857928 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 07:34:14.857955 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0925 07:34:14.858476 (+ 521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:34:14.858669 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:34:14.858672 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:34:14.858679 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:34:14.859128 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:14.859131 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:14.859746 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:14.859747 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:14.860443 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:14.860445 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:14.860727 (+ 282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:14.860728 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:14.860816 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:14.861370 (+ 554us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 07:34:14.861370 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:34:14.861395 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 07:34:14.861414 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:34:14.861671 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:34:14.861677 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:34:14.861680 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 07:34:14.861770 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 07:34:14.861859 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":85,"threads_started":1}
I20260925 07:34:14.862470 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790321392278483-9480-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790321392278483-9480-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790321392278483-9480-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790321392278483-9480-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.66.3:0
--local_ip_for_outbound_sockets=127.9.66.3
--webserver_interface=127.9.66.3
--webserver_port=0
--tserver_master_addrs=127.9.66.62:40053
--builtin_ntp_servers=127.9.66.20:35003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 07:34:14.862627 29516 heartbeater.cc:347] Connected to a master server at 127.9.66.62:40053
I20260925 07:34:14.862715 29516 heartbeater.cc:464] Registering TS with master...
I20260925 07:34:14.862877 29516 heartbeater.cc:511] Master 127.9.66.62:40053 requested a full tablet report, sending...
I20260925 07:34:14.863236 29197 ts_manager.cc:194] Registered new tserver with Master: ef1f64d47d80417c9fffa9df47efa71a (127.9.66.2:46377)
I20260925 07:34:14.863653 29197 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.66.2:46833
W20260925 07:34:14.939194 29519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:34:14.939389 29519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:34:14.939411 29519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:34:14.940467 29519 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:34:14.940501 29519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:34:14.940524 29519 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.66.3
I20260925 07:34:14.941795 29519 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:35003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790321392278483-9480-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790321392278483-9480-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.66.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790321392278483-9480-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.66.3
--webserver_port=0
--tserver_master_addrs=127.9.66.62:40053
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29519
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.66.3
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790321392278483-9480-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:34:14.941955 29519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:34:14.942255 29519 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:34:14.947328 29524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:34:14.947343 29525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:34:14.947333 29527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:34:14.947584 29519 server_base.cc:1034] running on GCE node
I20260925 07:34:14.947799 29519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:34:14.948035 29519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:34:14.949173 29519 hybrid_clock.cc:648] HybridClock initialized: now 1790321654949167 us; error 31 us; skew 500 ppm
I20260925 07:34:14.950523 29519 webserver.cc:460] Webserver started at http://127.9.66.3:43295/ using document root <none> and password file <none>
I20260925 07:34:14.950774 29519 fs_manager.cc:359] Metadata directory not provided
I20260925 07:34:14.950843 29519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:34:14.950984 29519 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:34:14.952041 29519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790321392278483-9480-0/minicluster-data/ts-2/data/instance:
uuid: "fb2a028239e346d28feb888546fe63be"
format_stamp: "Formatted at 2026-09-25 07:34:14 on dist-test-slave-57fn"
server_key: "e5bfea02ee33be23912193204547c371"
server_key_iv: "7e3047aeb8bd5f1b9f29b5c97bcbe5bc"
server_key_version: "encryptionkey@0"
I20260925 07:34:14.952387 29519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790321392278483-9480-0/minicluster-data/ts-2/wal/instance:
uuid: "fb2a028239e346d28feb888546fe63be"
format_stamp: "Formatted at 2026-09-25 07:34:14 on dist-test-slave-57fn"
server_key: "e5bfea02ee33be23912193204547c371"
server_key_iv: "7e3047aeb8bd5f1b9f29b5c97bcbe5bc"
server_key_version: "encryptionkey@0"
I20260925 07:34:14.953716 29519 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 07:34:14.954366 29533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:34:14.954532 29519 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 07:34:14.954620 29519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790321392278483-9480-0/minicluster-data/ts-2/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790321392278483-9480-0/minicluster-data/ts-2/wal
uuid: "fb2a028239e346d28feb888546fe63be"
format_stamp: "Formatted at 2026-09-25 07:34:14 on dist-test-slave-57fn"
server_key: "e5bfea02ee33be23912193204547c371"
server_key_iv: "7e3047aeb8bd5f1b9f29b5c97bcbe5bc"
server_key_version: "encryptionkey@0"
I20260925 07:34:14.954694 29519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790321392278483-9480-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790321392278483-9480-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790321392278483-9480-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:34:14.963601 29519 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:34:14.963841 29519 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:34:14.963986 29519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:34:14.964181 29519 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 07:34:14.964464 29519 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 07:34:14.964512 29519 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 07:34:14.964545 29519 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 07:34:14.964568 29519 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 07:34:14.970189 29519 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.3:41591
I20260925 07:34:14.970255 29646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.3:41591 every 8 connection(s)
I20260925 07:34:14.970542 29519 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790321392278483-9480-0/minicluster-data/ts-2/data/info.pb
I20260925 07:34:14.975598 29236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:14.971371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.3:37385 (local address 127.9.66.62:40053)
0925 07:34:14.971432 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:14.971434 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:14.971585 (+ 151us) server_negotiation.cc:407] Connection header received
0925 07:34:14.972122 (+ 537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:14.972124 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:14.972134 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:14.972197 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:34:14.972951 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:14.973456 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:14.974419 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:14.974596 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:14.974650 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:14.975413 (+ 763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:34:14.975414 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:34:14.975414 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 07:34:14.975418 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:34:14.975433 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 07:34:14.975436 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:34:14.975481 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:14.975527 (+ 46us) server_negotiation.cc:299] Negotiation successful
0925 07:34:14.975546 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 07:34:14.975833 29649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:14.971299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:40053 (local address 127.9.66.3:37385)
0925 07:34:14.971464 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 07:34:14.971501 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 07:34:14.972017 (+ 516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:34:14.972244 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:34:14.972248 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:34:14.972255 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:34:14.972812 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:14.972818 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:14.973571 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:14.973572 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:14.974329 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:14.974331 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:14.974667 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:14.974667 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:14.974766 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:14.975278 (+ 512us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 07:34:14.975278 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:34:14.975302 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 07:34:14.975322 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:34:14.975481 (+ 159us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:34:14.975483 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:34:14.975486 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 07:34:14.975528 (+ 42us) client_negotiation.cc:240] Negotiation successful
0925 07:34:14.975593 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":89,"threads_started":1}
I20260925 07:34:14.976315 29647 heartbeater.cc:347] Connected to a master server at 127.9.66.62:40053
I20260925 07:34:14.976408 29647 heartbeater.cc:464] Registering TS with master...
I20260925 07:34:14.976586 29647 heartbeater.cc:511] Master 127.9.66.62:40053 requested a full tablet report, sending...
I20260925 07:34:14.976985 29197 ts_manager.cc:194] Registered new tserver with Master: fb2a028239e346d28feb888546fe63be (127.9.66.3:41591)
I20260925 07:34:14.976999 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 29519
I20260925 07:34:14.977131 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790321392278483-9480-0/minicluster-data/ts-2/wal/instance
I20260925 07:34:14.977304 9480 external_mini_cluster.cc:1428] Setting key cf95c028c4199409bb0bb90a6f6de95b
I20260925 07:34:14.977407 29197 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.66.3:37385
I20260925 07:34:14.978205 9480 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 07:34:14.981874 29236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:14.979470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36028 (local address 127.9.66.62:40053)
0925 07:34:14.979502 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:14.979504 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:14.979630 (+ 126us) server_negotiation.cc:407] Connection header received
0925 07:34:14.979649 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:14.979649 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:14.979657 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:14.979693 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:34:14.979820 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:14.980197 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:14.980877 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:14.981055 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:14.981099 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:14.981670 (+ 571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:34:14.981671 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:34:14.981671 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 07:34:14.981675 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:34:14.981686 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 07:34:14.981689 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:34:14.981730 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:14.981779 (+ 49us) server_negotiation.cc:299] Negotiation successful
0925 07:34:14.981794 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 07:34:14.983330 29197 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36028:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 07:34:14.984911 29197 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 07:34:14.990974 29655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:14.987180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.3:41591 (local address 127.0.0.1:56582)
0925 07:34:14.987392 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0925 07:34:14.987403 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 07:34:14.987461 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:34:14.987797 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:34:14.987800 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:34:14.987801 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:34:14.987859 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:14.987861 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:14.988859 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:14.988860 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:14.989851 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:14.989854 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:14.990692 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:14.990693 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:14.990761 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:14.990774 (+ 13us) client_negotiation.cc:769] Sending connection context
0925 07:34:14.990841 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 07:34:14.990887 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":112,"threads_started":1}
I20260925 07:34:14.990983 29659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:14.987446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56582 (local address 127.9.66.3:41591)
0925 07:34:14.987603 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:14.987608 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:14.987617 (+ 9us) server_negotiation.cc:407] Connection header received
0925 07:34:14.987649 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:14.987651 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:14.987666 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:14.987933 (+ 267us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:34:14.987982 (+ 49us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:14.988762 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:14.989957 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:14.990625 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:14.990690 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:14.990767 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:14.990853 (+ 86us) server_negotiation.cc:299] Negotiation successful
0925 07:34:14.990893 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":89,"threads_started":1}
I20260925 07:34:14.991609 29658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:14.987296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.1:41079 (local address 127.0.0.1:60338)
0925 07:34:14.987679 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0925 07:34:14.987692 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 07:34:14.987751 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:34:14.987993 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:34:14.987994 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:34:14.987996 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:34:14.988049 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:14.988051 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:14.989381 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:14.989382 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:14.990375 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:14.990378 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:14.991375 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:14.991376 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:14.991443 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:14.991449 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 07:34:14.991503 (+ 54us) client_negotiation.cc:240] Negotiation successful
0925 07:34:14.991540 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"mutex_wait_us":1,"thread_start_us":67,"threads_started":1}
I20260925 07:34:14.991667 29660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:14.987500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60338 (local address 127.9.66.1:41079)
0925 07:34:14.987671 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:14.987675 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:14.987766 (+ 91us) server_negotiation.cc:407] Connection header received
0925 07:34:14.987814 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:14.987816 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:14.987833 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:14.987922 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:34:14.988118 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:14.989259 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:14.990483 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:14.991253 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:14.991336 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:14.991423 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:14.991518 (+ 95us) server_negotiation.cc:299] Negotiation successful
0925 07:34:14.991561 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":91,"threads_started":1}
I20260925 07:34:14.991787 29656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:14.987067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.2:46377 (local address 127.0.0.1:33654)
0925 07:34:14.987350 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0925 07:34:14.987391 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 07:34:14.987458 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:34:14.988159 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:34:14.988160 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:34:14.988162 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:34:14.988206 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:14.988207 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:14.989461 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:14.989462 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:14.990544 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:14.990546 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:14.991531 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:14.991533 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:14.991593 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:14.991597 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 07:34:14.991654 (+ 57us) client_negotiation.cc:240] Negotiation successful
0925 07:34:14.991699 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"mutex_wait_us":2,"thread_start_us":90,"threads_started":1}
I20260925 07:34:14.992028 29657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:14.987207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33654 (local address 127.9.66.2:46377)
0925 07:34:14.987951 (+ 744us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:14.987955 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:14.987966 (+ 11us) server_negotiation.cc:407] Connection header received
0925 07:34:14.988006 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:14.988008 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:14.988024 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:14.988096 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:34:14.988296 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:14.989375 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:14.990640 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:14.991329 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:14.991408 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:14.991491 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:14.991895 (+ 404us) server_negotiation.cc:299] Negotiation successful
0925 07:34:14.991944 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":624,"thread_start_us":81,"threads_started":1}
I20260925 07:34:14.991945 29581 tablet_service.cc:1467] Processing CreateTablet for tablet b9c0deb093154a5586778fab176de5ae (DEFAULT_TABLE table=test-table [id=2add5ad601134b48834b4d0baff7863b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:34:14.992285 29581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9c0deb093154a5586778fab176de5ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:34:14.992322 29319 tablet_service.cc:1467] Processing CreateTablet for tablet b9c0deb093154a5586778fab176de5ae (DEFAULT_TABLE table=test-table [id=2add5ad601134b48834b4d0baff7863b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:34:14.992589 29319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9c0deb093154a5586778fab176de5ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:34:14.992635 29450 tablet_service.cc:1467] Processing CreateTablet for tablet b9c0deb093154a5586778fab176de5ae (DEFAULT_TABLE table=test-table [id=2add5ad601134b48834b4d0baff7863b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 07:34:14.992942 29450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9c0deb093154a5586778fab176de5ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:34:14.995095 29661 tablet_bootstrap.cc:492] T b9c0deb093154a5586778fab176de5ae P 890ac1f3a91c469fa2a89a046a7d3786: Bootstrap starting.
I20260925 07:34:14.995769 29661 tablet_bootstrap.cc:654] T b9c0deb093154a5586778fab176de5ae P 890ac1f3a91c469fa2a89a046a7d3786: Neither blocks nor log segments found. Creating new log.
I20260925 07:34:14.996068 29662 tablet_bootstrap.cc:492] T b9c0deb093154a5586778fab176de5ae P fb2a028239e346d28feb888546fe63be: Bootstrap starting.
I20260925 07:34:14.996133 29661 log.cc:824] T b9c0deb093154a5586778fab176de5ae P 890ac1f3a91c469fa2a89a046a7d3786: Log is configured to *not* fsync() on all Append() calls
I20260925 07:34:14.996433 29663 tablet_bootstrap.cc:492] T b9c0deb093154a5586778fab176de5ae P ef1f64d47d80417c9fffa9df47efa71a: Bootstrap starting.
I20260925 07:34:14.996783 29662 tablet_bootstrap.cc:654] T b9c0deb093154a5586778fab176de5ae P fb2a028239e346d28feb888546fe63be: Neither blocks nor log segments found. Creating new log.
I20260925 07:34:14.996889 29661 tablet_bootstrap.cc:492] T b9c0deb093154a5586778fab176de5ae P 890ac1f3a91c469fa2a89a046a7d3786: No bootstrap required, opened a new log
I20260925 07:34:14.996978 29661 ts_tablet_manager.cc:1397] T b9c0deb093154a5586778fab176de5ae P 890ac1f3a91c469fa2a89a046a7d3786: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 07:34:14.997074 29663 tablet_bootstrap.cc:654] T b9c0deb093154a5586778fab176de5ae P ef1f64d47d80417c9fffa9df47efa71a: Neither blocks nor log segments found. Creating new log.
I20260925 07:34:14.997140 29662 log.cc:824] T b9c0deb093154a5586778fab176de5ae P fb2a028239e346d28feb888546fe63be: Log is configured to *not* fsync() on all Append() calls
I20260925 07:34:14.997393 29663 log.cc:824] T b9c0deb093154a5586778fab176de5ae P ef1f64d47d80417c9fffa9df47efa71a: Log is configured to *not* fsync() on all Append() calls
I20260925 07:34:14.997925 29662 tablet_bootstrap.cc:492] T b9c0deb093154a5586778fab176de5ae P fb2a028239e346d28feb888546fe63be: No bootstrap required, opened a new log
I20260925 07:34:14.998029 29662 ts_tablet_manager.cc:1397] T b9c0deb093154a5586778fab176de5ae P fb2a028239e346d28feb888546fe63be: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 07:34:14.999122 29661 raft_consensus.cc:348] T b9c0deb093154a5586778fab176de5ae P 890ac1f3a91c469fa2a89a046a7d3786 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef1f64d47d80417c9fffa9df47efa71a" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 46377 } } peers { permanent_uuid: "890ac1f3a91c469fa2a89a046a7d3786" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 41079 } } peers { permanent_uuid: "fb2a028239e346d28feb888546fe63be" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 41591 } }
I20260925 07:34:14.999310 29661 raft_consensus.cc:374] T b9c0deb093154a5586778fab176de5ae P 890ac1f3a91c469fa2a89a046a7d3786 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:34:14.999357 29661 raft_consensus.cc:729] T b9c0deb093154a5586778fab176de5ae P 890ac1f3a91c469fa2a89a046a7d3786 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 890ac1f3a91c469fa2a89a046a7d3786, State: Initialized, Role: FOLLOWER
I20260925 07:34:14.999472 29661 consensus_queue.cc:260] T b9c0deb093154a5586778fab176de5ae P 890ac1f3a91c469fa2a89a046a7d3786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef1f64d47d80417c9fffa9df47efa71a" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 46377 } } peers { permanent_uuid: "890ac1f3a91c469fa2a89a046a7d3786" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 41079 } } peers { permanent_uuid: "fb2a028239e346d28feb888546fe63be" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 41591 } }
I20260925 07:34:14.999725 29661 ts_tablet_manager.cc:1428] T b9c0deb093154a5586778fab176de5ae P 890ac1f3a91c469fa2a89a046a7d3786: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 07:34:14.999759 29385 heartbeater.cc:503] Master 127.9.66.62:40053 was elected leader, sending a full tablet report...
I20260925 07:34:15.000233 29662 raft_consensus.cc:348] T b9c0deb093154a5586778fab176de5ae P fb2a028239e346d28feb888546fe63be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef1f64d47d80417c9fffa9df47efa71a" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 46377 } } peers { permanent_uuid: "890ac1f3a91c469fa2a89a046a7d3786" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 41079 } } peers { permanent_uuid: "fb2a028239e346d28feb888546fe63be" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 41591 } }
I20260925 07:34:15.000399 29662 raft_consensus.cc:374] T b9c0deb093154a5586778fab176de5ae P fb2a028239e346d28feb888546fe63be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:34:15.000437 29662 raft_consensus.cc:729] T b9c0deb093154a5586778fab176de5ae P fb2a028239e346d28feb888546fe63be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb2a028239e346d28feb888546fe63be, State: Initialized, Role: FOLLOWER
I20260925 07:34:15.000602 29662 consensus_queue.cc:260] T b9c0deb093154a5586778fab176de5ae P fb2a028239e346d28feb888546fe63be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef1f64d47d80417c9fffa9df47efa71a" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 46377 } } peers { permanent_uuid: "890ac1f3a91c469fa2a89a046a7d3786" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 41079 } } peers { permanent_uuid: "fb2a028239e346d28feb888546fe63be" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 41591 } }
I20260925 07:34:15.000893 29662 ts_tablet_manager.cc:1428] T b9c0deb093154a5586778fab176de5ae P fb2a028239e346d28feb888546fe63be: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 07:34:15.000941 29647 heartbeater.cc:503] Master 127.9.66.62:40053 was elected leader, sending a full tablet report...
I20260925 07:34:15.001590 29663 tablet_bootstrap.cc:492] T b9c0deb093154a5586778fab176de5ae P ef1f64d47d80417c9fffa9df47efa71a: No bootstrap required, opened a new log
I20260925 07:34:15.001662 29663 ts_tablet_manager.cc:1397] T b9c0deb093154a5586778fab176de5ae P ef1f64d47d80417c9fffa9df47efa71a: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260925 07:34:15.003428 29663 raft_consensus.cc:348] T b9c0deb093154a5586778fab176de5ae P ef1f64d47d80417c9fffa9df47efa71a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef1f64d47d80417c9fffa9df47efa71a" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 46377 } } peers { permanent_uuid: "890ac1f3a91c469fa2a89a046a7d3786" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 41079 } } peers { permanent_uuid: "fb2a028239e346d28feb888546fe63be" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 41591 } }
I20260925 07:34:15.003564 29663 raft_consensus.cc:374] T b9c0deb093154a5586778fab176de5ae P ef1f64d47d80417c9fffa9df47efa71a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:34:15.003608 29663 raft_consensus.cc:729] T b9c0deb093154a5586778fab176de5ae P ef1f64d47d80417c9fffa9df47efa71a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef1f64d47d80417c9fffa9df47efa71a, State: Initialized, Role: FOLLOWER
I20260925 07:34:15.003720 29663 consensus_queue.cc:260] T b9c0deb093154a5586778fab176de5ae P ef1f64d47d80417c9fffa9df47efa71a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef1f64d47d80417c9fffa9df47efa71a" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 46377 } } peers { permanent_uuid: "890ac1f3a91c469fa2a89a046a7d3786" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 41079 } } peers { permanent_uuid: "fb2a028239e346d28feb888546fe63be" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 41591 } }
I20260925 07:34:15.003960 29663 ts_tablet_manager.cc:1428] T b9c0deb093154a5586778fab176de5ae P ef1f64d47d80417c9fffa9df47efa71a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 07:34:15.004053 29516 heartbeater.cc:503] Master 127.9.66.62:40053 was elected leader, sending a full tablet report...
I20260925 07:34:15.036847 29669 raft_consensus.cc:482] T b9c0deb093154a5586778fab176de5ae P ef1f64d47d80417c9fffa9df47efa71a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 07:34:15.036929 29669 raft_consensus.cc:504] T b9c0deb093154a5586778fab176de5ae P ef1f64d47d80417c9fffa9df47efa71a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef1f64d47d80417c9fffa9df47efa71a" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 46377 } } peers { permanent_uuid: "890ac1f3a91c469fa2a89a046a7d3786" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 41079 } } peers { permanent_uuid: "fb2a028239e346d28feb888546fe63be" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 41591 } }
I20260925 07:34:15.037204 29669 leader_election.cc:288] T b9c0deb093154a5586778fab176de5ae P ef1f64d47d80417c9fffa9df47efa71a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 890ac1f3a91c469fa2a89a046a7d3786 (127.9.66.1:41079), fb2a028239e346d28feb888546fe63be (127.9.66.3:41591)
I20260925 07:34:15.041007 29659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:15.037605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.2:36935 (local address 127.9.66.3:41591)
0925 07:34:15.037666 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:15.037668 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:15.037676 (+ 8us) server_negotiation.cc:407] Connection header received
0925 07:34:15.037701 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:15.037702 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:15.037710 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:15.037762 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:34:15.037911 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:15.038598 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:15.040130 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:15.040675 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:15.040742 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:15.040751 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:15.040912 (+ 161us) server_negotiation.cc:299] Negotiation successful
0925 07:34:15.040946 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 07:34:15.041007 29660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:15.037481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.66.2:52619 (local address 127.9.66.1:41079)
0925 07:34:15.037542 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:15.037544 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:15.037561 (+ 17us) server_negotiation.cc:407] Connection header received
0925 07:34:15.037587 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:15.037588 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:15.037597 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:15.037672 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 07:34:15.037841 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:15.038597 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:15.040130 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:15.040670 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:15.040742 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:15.040751 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:15.040913 (+ 162us) server_negotiation.cc:299] Negotiation successful
0925 07:34:15.040947 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 07:34:15.041044 29518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:15.037359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.1:41079 (local address 127.9.66.2:52619)
0925 07:34:15.037451 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0925 07:34:15.037465 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 07:34:15.037533 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:34:15.037676 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:34:15.037677 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:34:15.037678 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:34:15.037763 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:15.037765 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:15.038717 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:15.038719 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:15.040038 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:15.040041 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:15.040756 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:15.040757 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:15.040830 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:15.040834 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 07:34:15.040890 (+ 56us) client_negotiation.cc:240] Negotiation successful
0925 07:34:15.040936 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72,"mutex_wait_us":38}
I20260925 07:34:15.041050 29670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:15.037359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.3:41591 (local address 127.9.66.2:36935)
0925 07:34:15.037542 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0925 07:34:15.037553 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 07:34:15.037603 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:34:15.037783 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:34:15.037783 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:34:15.037785 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 07:34:15.037847 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:15.037848 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:15.038717 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:15.038719 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:15.040051 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:15.040053 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:15.040768 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:15.040769 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:15.040830 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:15.040834 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 07:34:15.040890 (+ 56us) client_negotiation.cc:240] Negotiation successful
0925 07:34:15.040936 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"mutex_wait_us":81,"thread_start_us":97,"threads_started":1}
I20260925 07:34:15.041432 29601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9c0deb093154a5586778fab176de5ae" candidate_uuid: "ef1f64d47d80417c9fffa9df47efa71a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb2a028239e346d28feb888546fe63be" is_pre_election: true
I20260925 07:34:15.041471 29339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9c0deb093154a5586778fab176de5ae" candidate_uuid: "ef1f64d47d80417c9fffa9df47efa71a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "890ac1f3a91c469fa2a89a046a7d3786" is_pre_election: true
I20260925 07:34:15.041589 29601 raft_consensus.cc:2454] T b9c0deb093154a5586778fab176de5ae P fb2a028239e346d28feb888546fe63be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef1f64d47d80417c9fffa9df47efa71a in term 0.
I20260925 07:34:15.041639 29339 raft_consensus.cc:2454] T b9c0deb093154a5586778fab176de5ae P 890ac1f3a91c469fa2a89a046a7d3786 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef1f64d47d80417c9fffa9df47efa71a in term 0.
I20260925 07:34:15.041832 29406 leader_election.cc:302] T b9c0deb093154a5586778fab176de5ae P ef1f64d47d80417c9fffa9df47efa71a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ef1f64d47d80417c9fffa9df47efa71a, fb2a028239e346d28feb888546fe63be; no voters:
I20260925 07:34:15.041972 29669 raft_consensus.cc:2781] T b9c0deb093154a5586778fab176de5ae P ef1f64d47d80417c9fffa9df47efa71a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 07:34:15.042025 29669 raft_consensus.cc:482] T b9c0deb093154a5586778fab176de5ae P ef1f64d47d80417c9fffa9df47efa71a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 07:34:15.042047 29669 raft_consensus.cc:3037] T b9c0deb093154a5586778fab176de5ae P ef1f64d47d80417c9fffa9df47efa71a [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:34:15.042760 29669 raft_consensus.cc:504] T b9c0deb093154a5586778fab176de5ae P ef1f64d47d80417c9fffa9df47efa71a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef1f64d47d80417c9fffa9df47efa71a" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 46377 } } peers { permanent_uuid: "890ac1f3a91c469fa2a89a046a7d3786" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 41079 } } peers { permanent_uuid: "fb2a028239e346d28feb888546fe63be" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 41591 } }
I20260925 07:34:15.043004 29669 leader_election.cc:288] T b9c0deb093154a5586778fab176de5ae P ef1f64d47d80417c9fffa9df47efa71a [CANDIDATE]: Term 1 election: Requested vote from peers 890ac1f3a91c469fa2a89a046a7d3786 (127.9.66.1:41079), fb2a028239e346d28feb888546fe63be (127.9.66.3:41591)
I20260925 07:34:15.043092 29339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9c0deb093154a5586778fab176de5ae" candidate_uuid: "ef1f64d47d80417c9fffa9df47efa71a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "890ac1f3a91c469fa2a89a046a7d3786"
I20260925 07:34:15.043140 29601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9c0deb093154a5586778fab176de5ae" candidate_uuid: "ef1f64d47d80417c9fffa9df47efa71a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb2a028239e346d28feb888546fe63be"
I20260925 07:34:15.043176 29339 raft_consensus.cc:3037] T b9c0deb093154a5586778fab176de5ae P 890ac1f3a91c469fa2a89a046a7d3786 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:34:15.043208 29601 raft_consensus.cc:3037] T b9c0deb093154a5586778fab176de5ae P fb2a028239e346d28feb888546fe63be [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:34:15.044000 29339 raft_consensus.cc:2454] T b9c0deb093154a5586778fab176de5ae P 890ac1f3a91c469fa2a89a046a7d3786 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef1f64d47d80417c9fffa9df47efa71a in term 1.
I20260925 07:34:15.044000 29601 raft_consensus.cc:2454] T b9c0deb093154a5586778fab176de5ae P fb2a028239e346d28feb888546fe63be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef1f64d47d80417c9fffa9df47efa71a in term 1.
I20260925 07:34:15.044176 29406 leader_election.cc:302] T b9c0deb093154a5586778fab176de5ae P ef1f64d47d80417c9fffa9df47efa71a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ef1f64d47d80417c9fffa9df47efa71a, fb2a028239e346d28feb888546fe63be; no voters:
I20260925 07:34:15.044283 29669 raft_consensus.cc:2781] T b9c0deb093154a5586778fab176de5ae P ef1f64d47d80417c9fffa9df47efa71a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:34:15.044440 29669 raft_consensus.cc:686] T b9c0deb093154a5586778fab176de5ae P ef1f64d47d80417c9fffa9df47efa71a [term 1 LEADER]: Becoming Leader. State: Replica: ef1f64d47d80417c9fffa9df47efa71a, State: Running, Role: LEADER
I20260925 07:34:15.044517 29669 consensus_queue.cc:237] T b9c0deb093154a5586778fab176de5ae P ef1f64d47d80417c9fffa9df47efa71a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef1f64d47d80417c9fffa9df47efa71a" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 46377 } } peers { permanent_uuid: "890ac1f3a91c469fa2a89a046a7d3786" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 41079 } } peers { permanent_uuid: "fb2a028239e346d28feb888546fe63be" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 41591 } }
I20260925 07:34:15.045238 29197 catalog_manager.cc:5495] T b9c0deb093154a5586778fab176de5ae P ef1f64d47d80417c9fffa9df47efa71a reported cstate change: term changed from 0 to 1, leader changed from <none> to ef1f64d47d80417c9fffa9df47efa71a (127.9.66.2). New cstate: current_term: 1 leader_uuid: "ef1f64d47d80417c9fffa9df47efa71a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef1f64d47d80417c9fffa9df47efa71a" member_type: VOTER last_known_addr { host: "127.9.66.2" port: 46377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "890ac1f3a91c469fa2a89a046a7d3786" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 41079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb2a028239e346d28feb888546fe63be" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 41591 } health_report { overall_health: UNKNOWN } } }
I20260925 07:34:15.050930 29657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:15.048652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33660 (local address 127.9.66.2:46377)
0925 07:34:15.048689 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:15.048691 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:15.048699 (+ 8us) server_negotiation.cc:407] Connection header received
0925 07:34:15.048748 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:15.048751 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:15.048769 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:15.048839 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 07:34:15.048936 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:15.049345 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:15.050162 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:15.050334 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:15.050405 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:15.050581 (+ 176us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 07:34:15.050660 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 07:34:15.050762 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:15.050798 (+ 36us) server_negotiation.cc:299] Negotiation successful
0925 07:34:15.050816 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 07:34:15.052485 29601 raft_consensus.cc:1261] T b9c0deb093154a5586778fab176de5ae P fb2a028239e346d28feb888546fe63be [term 1 FOLLOWER]: Refusing update from remote peer ef1f64d47d80417c9fffa9df47efa71a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:34:15.052485 29339 raft_consensus.cc:1261] T b9c0deb093154a5586778fab176de5ae P 890ac1f3a91c469fa2a89a046a7d3786 [term 1 FOLLOWER]: Refusing update from remote peer ef1f64d47d80417c9fffa9df47efa71a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:34:15.052811 29669 consensus_queue.cc:1035] T b9c0deb093154a5586778fab176de5ae P ef1f64d47d80417c9fffa9df47efa71a [LEADER]: Connected to new peer: Peer: permanent_uuid: "890ac1f3a91c469fa2a89a046a7d3786" member_type: VOTER last_known_addr { host: "127.9.66.1" port: 41079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:34:15.052979 29671 consensus_queue.cc:1035] T b9c0deb093154a5586778fab176de5ae P ef1f64d47d80417c9fffa9df47efa71a [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb2a028239e346d28feb888546fe63be" member_type: VOTER last_known_addr { host: "127.9.66.3" port: 41591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:34:15.055392 29675 mvcc.cc:205] Tried to move back new op lower bound from 7333157499092615168 to 7333157499063066624. Current Snapshot: MvccSnapshot[applied={T|T < 7333157499092615168 or (T in {7333157499092615168})}]
I20260925 07:34:15.055493 29676 mvcc.cc:205] Tried to move back new op lower bound from 7333157499092615168 to 7333157499063066624. Current Snapshot: MvccSnapshot[applied={T|T < 7333157499092615168}]
I20260925 07:34:15.057776 29196 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36028:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 07:34:15.057875 29196 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36028:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 07:34:15.058568 29196 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 46c2640c72a94770b5333ac0a44b551e: Sending DeleteTablet for 3 replicas of tablet b9c0deb093154a5586778fab176de5ae
I20260925 07:34:15.058897 29319 tablet_service.cc:1514] Processing DeleteTablet for tablet b9c0deb093154a5586778fab176de5ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:34:15 UTC) from {username='slave'} at 127.0.0.1:60338
I20260925 07:34:15.058974 29450 tablet_service.cc:1514] Processing DeleteTablet for tablet b9c0deb093154a5586778fab176de5ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:34:15 UTC) from {username='slave'} at 127.0.0.1:33654
I20260925 07:34:15.059091 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 29257
I20260925 07:34:15.059111 29581 tablet_service.cc:1514] Processing DeleteTablet for tablet b9c0deb093154a5586778fab176de5ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 07:34:15 UTC) from {username='slave'} at 127.0.0.1:56582
I20260925 07:34:15.059788 29684 tablet_replica.cc:330] T b9c0deb093154a5586778fab176de5ae P ef1f64d47d80417c9fffa9df47efa71a: stopping tablet replica
I20260925 07:34:15.059876 29684 raft_consensus.cc:2229] T b9c0deb093154a5586778fab176de5ae P ef1f64d47d80417c9fffa9df47efa71a [term 1 LEADER]: Raft consensus shutting down.
I20260925 07:34:15.059993 29684 raft_consensus.cc:2258] T b9c0deb093154a5586778fab176de5ae P ef1f64d47d80417c9fffa9df47efa71a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 07:34:15.060372 29684 ts_tablet_manager.cc:1905] T b9c0deb093154a5586778fab176de5ae P ef1f64d47d80417c9fffa9df47efa71a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 07:34:15.062294 29684 ts_tablet_manager.cc:1918] T b9c0deb093154a5586778fab176de5ae P ef1f64d47d80417c9fffa9df47efa71a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 07:34:15.062354 29684 log.cc:1196] T b9c0deb093154a5586778fab176de5ae P ef1f64d47d80417c9fffa9df47efa71a: Deleting WAL directory at /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790321392278483-9480-0/minicluster-data/ts-1/wal/wals/b9c0deb093154a5586778fab176de5ae
I20260925 07:34:15.062625 29684 ts_tablet_manager.cc:1939] T b9c0deb093154a5586778fab176de5ae P ef1f64d47d80417c9fffa9df47efa71a: Deleting consensus metadata
I20260925 07:34:15.063043 29685 tablet_replica.cc:330] T b9c0deb093154a5586778fab176de5ae P fb2a028239e346d28feb888546fe63be: stopping tablet replica
I20260925 07:34:15.063057 29182 catalog_manager.cc:4841] TS ef1f64d47d80417c9fffa9df47efa71a (127.9.66.2:46377): tablet b9c0deb093154a5586778fab176de5ae (table test-table [id=2add5ad601134b48834b4d0baff7863b]) successfully deleted
I20260925 07:34:15.063210 29685 raft_consensus.cc:2229] T b9c0deb093154a5586778fab176de5ae P fb2a028239e346d28feb888546fe63be [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 07:34:15.063331 29685 raft_consensus.cc:2258] T b9c0deb093154a5586778fab176de5ae P fb2a028239e346d28feb888546fe63be [term 1 FOLLOWER]: Raft consensus is shut down!
W20260925 07:34:15.066442 29182 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.66.1:41079 (error 108)
I20260925 07:34:15.066736 29685 ts_tablet_manager.cc:1905] T b9c0deb093154a5586778fab176de5ae P fb2a028239e346d28feb888546fe63be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 07:34:15.066900 29656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:15.066752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.1:41079 (local address 127.0.0.1:60354)
0925 07:34:15.066812 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0925 07:34:15.066854 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.1:41079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260925 07:34:15.067080 29182 catalog_manager.cc:4584] TS 890ac1f3a91c469fa2a89a046a7d3786 (127.9.66.1:41079): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b9c0deb093154a5586778fab176de5ae: Network error: Client connection negotiation failed: client connection to 127.9.66.1:41079: connect: Connection refused (error 111)
I20260925 07:34:15.068005 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 29388
I20260925 07:34:15.068706 29685 ts_tablet_manager.cc:1918] T b9c0deb093154a5586778fab176de5ae P fb2a028239e346d28feb888546fe63be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 07:34:15.068774 29685 log.cc:1196] T b9c0deb093154a5586778fab176de5ae P fb2a028239e346d28feb888546fe63be: Deleting WAL directory at /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790321392278483-9480-0/minicluster-data/ts-2/wal/wals/b9c0deb093154a5586778fab176de5ae
I20260925 07:34:15.069024 29685 ts_tablet_manager.cc:1939] T b9c0deb093154a5586778fab176de5ae P fb2a028239e346d28feb888546fe63be: Deleting consensus metadata
I20260925 07:34:15.069410 29184 catalog_manager.cc:4841] TS fb2a028239e346d28feb888546fe63be (127.9.66.3:41591): tablet b9c0deb093154a5586778fab176de5ae (table test-table [id=2add5ad601134b48834b4d0baff7863b]) successfully deleted
I20260925 07:34:15.072890 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 29519
I20260925 07:34:15.077378 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 29166
2026-09-25T07:34:15Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (651 ms)
[----------] 6 tests from EncryptionPolicyTest (7966 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790321392278483-9480-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 07:34:15 dist-test-slave-57fn krb5kdc[29689](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 07:34:15 dist-test-slave-57fn krb5kdc[29689](info): set up 2 sockets
Sep 25 07:34:15 dist-test-slave-57fn krb5kdc[29689](info): commencing operation
krb5kdc: starting...
W20260925 07:34:16.089313 9480 mini_kdc.cc:121] Time spent starting KDC: real 0.993s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790321392278483-9480-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790321392278483-9480-0/krb5kdc/test-user.keytab.
Sep 25 07:34:16 dist-test-slave-57fn krb5kdc[29689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321656, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T07:34:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T07:34:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790321392278483-9480-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790321392278483-9480-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 07:34:16.201357 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790321392278483-9480-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790321392278483-9480-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790321392278483-9480-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46159
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790321392278483-9480-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46159
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:34:16.279237 29702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:34:16.279421 29702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:34:16.279443 29702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:34:16.280534 29702 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:34:16.280572 29702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:34:16.280586 29702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:34:16.280635 29702 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:34:16.280651 29702 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:34:16.282099 29702 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790321392278483-9480-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790321392278483-9480-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46159
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790321392278483-9480-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46159
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29702
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790321392278483-9480-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:34:16.282343 29702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:34:16.282559 29702 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:34:16.287154 29710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:34:16.287236 29707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:34:16.287149 29708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:34:16.287286 29702 server_base.cc:1034] running on GCE node
I20260925 07:34:16.287519 29702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:34:16.287817 29702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:34:16.288980 29702 hybrid_clock.cc:648] HybridClock initialized: now 1790321656288970 us; error 48 us; skew 500 ppm
Sep 25 07:34:16 dist-test-slave-57fn krb5kdc[29689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321656, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:34:16.291318 29702 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 07:34:16.291739 29702 webserver.cc:460] Webserver started at http://127.0.0.1:40749/ using document root <none> and password file <none>
I20260925 07:34:16.291908 29702 fs_manager.cc:359] Metadata directory not provided
I20260925 07:34:16.291945 29702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:34:16.292024 29702 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:34:16.292915 29702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790321392278483-9480-0/minicluster-data/master-0/data/instance:
uuid: "a131459a3e7c4a8f93298e2a83bc30c9"
format_stamp: "Formatted at 2026-09-25 07:34:16 on dist-test-slave-57fn"
server_key: "c8f17f1b80ced0f308b3bac197e1ded9"
server_key_iv: "7ebf072e643f12d9aed8317170af9760"
server_key_version: "encryptionkey@0"
I20260925 07:34:16.293195 29702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790321392278483-9480-0/minicluster-data/master-0/wal/instance:
uuid: "a131459a3e7c4a8f93298e2a83bc30c9"
format_stamp: "Formatted at 2026-09-25 07:34:16 on dist-test-slave-57fn"
server_key: "c8f17f1b80ced0f308b3bac197e1ded9"
server_key_iv: "7ebf072e643f12d9aed8317170af9760"
server_key_version: "encryptionkey@0"
I20260925 07:34:16.294525 29702 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 07:34:16.295197 29717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:34:16.295346 29702 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 07:34:16.295430 29702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790321392278483-9480-0/minicluster-data/master-0/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790321392278483-9480-0/minicluster-data/master-0/wal
uuid: "a131459a3e7c4a8f93298e2a83bc30c9"
format_stamp: "Formatted at 2026-09-25 07:34:16 on dist-test-slave-57fn"
server_key: "c8f17f1b80ced0f308b3bac197e1ded9"
server_key_iv: "7ebf072e643f12d9aed8317170af9760"
server_key_version: "encryptionkey@0"
I20260925 07:34:16.295483 29702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790321392278483-9480-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790321392278483-9480-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790321392278483-9480-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:34:16.309580 29702 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:34:16.311164 29702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:34:16.311261 29702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:34:16.315100 29702 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46159
I20260925 07:34:16.315162 29769 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46159 every 8 connection(s)
I20260925 07:34:16.315483 29702 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
I20260925 07:34:16.316526 29770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:34:16.319060 29770 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a131459a3e7c4a8f93298e2a83bc30c9: Bootstrap starting.
I20260925 07:34:16.319669 29770 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a131459a3e7c4a8f93298e2a83bc30c9: Neither blocks nor log segments found. Creating new log.
I20260925 07:34:16.319998 29770 log.cc:824] T 00000000000000000000000000000000 P a131459a3e7c4a8f93298e2a83bc30c9: Log is configured to *not* fsync() on all Append() calls
I20260925 07:34:16.320711 29770 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a131459a3e7c4a8f93298e2a83bc30c9: No bootstrap required, opened a new log
I20260925 07:34:16.322644 29770 raft_consensus.cc:348] T 00000000000000000000000000000000 P a131459a3e7c4a8f93298e2a83bc30c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a131459a3e7c4a8f93298e2a83bc30c9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46159 } }
I20260925 07:34:16.322809 29770 raft_consensus.cc:374] T 00000000000000000000000000000000 P a131459a3e7c4a8f93298e2a83bc30c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:34:16.322880 29770 raft_consensus.cc:729] T 00000000000000000000000000000000 P a131459a3e7c4a8f93298e2a83bc30c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a131459a3e7c4a8f93298e2a83bc30c9, State: Initialized, Role: FOLLOWER
I20260925 07:34:16.323061 29770 consensus_queue.cc:260] T 00000000000000000000000000000000 P a131459a3e7c4a8f93298e2a83bc30c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a131459a3e7c4a8f93298e2a83bc30c9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46159 } }
I20260925 07:34:16.323189 29770 raft_consensus.cc:388] T 00000000000000000000000000000000 P a131459a3e7c4a8f93298e2a83bc30c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 07:34:16.323256 29770 raft_consensus.cc:482] T 00000000000000000000000000000000 P a131459a3e7c4a8f93298e2a83bc30c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 07:34:16.323309 29770 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a131459a3e7c4a8f93298e2a83bc30c9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:34:16.323896 29770 raft_consensus.cc:504] T 00000000000000000000000000000000 P a131459a3e7c4a8f93298e2a83bc30c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a131459a3e7c4a8f93298e2a83bc30c9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46159 } }
I20260925 07:34:16.324020 29770 leader_election.cc:302] T 00000000000000000000000000000000 P a131459a3e7c4a8f93298e2a83bc30c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a131459a3e7c4a8f93298e2a83bc30c9; no voters:
I20260925 07:34:16.324208 29770 leader_election.cc:288] T 00000000000000000000000000000000 P a131459a3e7c4a8f93298e2a83bc30c9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 07:34:16.324271 29773 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a131459a3e7c4a8f93298e2a83bc30c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:34:16.324481 29770 sys_catalog.cc:564] T 00000000000000000000000000000000 P a131459a3e7c4a8f93298e2a83bc30c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:34:16.324499 29773 raft_consensus.cc:686] T 00000000000000000000000000000000 P a131459a3e7c4a8f93298e2a83bc30c9 [term 1 LEADER]: Becoming Leader. State: Replica: a131459a3e7c4a8f93298e2a83bc30c9, State: Running, Role: LEADER
I20260925 07:34:16.324580 29773 consensus_queue.cc:237] T 00000000000000000000000000000000 P a131459a3e7c4a8f93298e2a83bc30c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a131459a3e7c4a8f93298e2a83bc30c9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46159 } }
I20260925 07:34:16.324939 29775 sys_catalog.cc:455] T 00000000000000000000000000000000 P a131459a3e7c4a8f93298e2a83bc30c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a131459a3e7c4a8f93298e2a83bc30c9. Latest consensus state: current_term: 1 leader_uuid: "a131459a3e7c4a8f93298e2a83bc30c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a131459a3e7c4a8f93298e2a83bc30c9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46159 } } }
I20260925 07:34:16.325026 29775 sys_catalog.cc:458] T 00000000000000000000000000000000 P a131459a3e7c4a8f93298e2a83bc30c9 [sys.catalog]: This master's current role is: LEADER
I20260925 07:34:16.325276 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 29702
I20260925 07:34:16.325388 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790321392278483-9480-0/minicluster-data/master-0/wal/instance
I20260925 07:34:16.325668 9480 external_mini_cluster.cc:1428] Setting key e2db5531aae4fad9229990ebbdcbf4f3
I20260925 07:34:16.325143 29774 sys_catalog.cc:455] T 00000000000000000000000000000000 P a131459a3e7c4a8f93298e2a83bc30c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a131459a3e7c4a8f93298e2a83bc30c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a131459a3e7c4a8f93298e2a83bc30c9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46159 } } }
I20260925 07:34:16.325773 29774 sys_catalog.cc:458] T 00000000000000000000000000000000 P a131459a3e7c4a8f93298e2a83bc30c9 [sys.catalog]: This master's current role is: LEADER
W20260925 07:34:16.326574 29789 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a131459a3e7c4a8f93298e2a83bc30c9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 07:34:16.326638 29789 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 07:34:16.326716 29787 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 07:34:16.326898 29787 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 07:34:16.328505 29787 catalog_manager.cc:1347] Generated new cluster ID: 2c0fe2d04c9140ac8e0a13e1b56678e0
I20260925 07:34:16.328564 29787 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 07:34:16 dist-test-slave-57fn krb5kdc[29689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321656, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 07:34:16.335537 29791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:16.326118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58159 (local address 127.0.0.1:46159)
0925 07:34:16.326316 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:16.326325 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:16.326502 (+ 177us) server_negotiation.cc:407] Connection header received
0925 07:34:16.326674 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:16.326682 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:16.326737 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:16.326863 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:34:16.327563 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:16.328656 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:16.329408 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:16.329920 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:16.330043 (+ 123us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:16.332276 (+ 2233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:34:16.332280 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:34:16.332293 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:34:16.332312 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:34:16.333968 (+ 1656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:34:16.334350 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:34:16.334353 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:34:16.334353 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:34:16.334450 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:34:16.334701 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:34:16.334702 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:34:16.334702 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:34:16.334963 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:34:16.335142 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:16.335253 (+ 111us) server_negotiation.cc:299] Negotiation successful
0925 07:34:16.335357 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":71,"threads_started":1}
I20260925 07:34:16.336462 9480 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 07:34:16.347599 29791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:16.337823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48467 (local address 127.0.0.1:46159)
0925 07:34:16.337865 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:16.337867 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:16.341714 (+ 3847us) server_negotiation.cc:407] Connection header received
0925 07:34:16.341784 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:16.341785 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:16.341795 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:16.341848 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:34:16.342342 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:16.342837 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:16.343548 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:16.343753 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:16.343837 (+ 84us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:16.344976 (+ 1139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:34:16.344978 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:34:16.344978 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:34:16.344985 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:34:16.346427 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:34:16.346782 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:34:16.346783 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:34:16.346784 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:34:16.346835 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:34:16.347082 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:34:16.347083 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:34:16.347083 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:34:16.347206 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:34:16.347313 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:16.347446 (+ 133us) server_negotiation.cc:299] Negotiation successful
0925 07:34:16.347507 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 07:34:16.376956 29787 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 07:34:16.377561 29787 catalog_manager.cc:1504] Loading token signing keys...
I20260925 07:34:16.383219 29787 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a131459a3e7c4a8f93298e2a83bc30c9: Generated new TSK 0
I20260925 07:34:16.383448 29787 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 07:34:16.387279 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 29702
W20260925 07:34:16.391633 29687 connection.cc:535] client connection to 127.0.0.1:46159 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46159): Connection reset by peer (error 104)
2026-09-25T07:34:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1308 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790321392278483-9480-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 07:34:16 dist-test-slave-57fn krb5kdc[29801](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 07:34:16 dist-test-slave-57fn krb5kdc[29801](info): set up 2 sockets
Sep 25 07:34:16 dist-test-slave-57fn krb5kdc[29801](info): commencing operation
krb5kdc: starting...
W20260925 07:34:17.413590 9480 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790321392278483-9480-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790321392278483-9480-0/krb5kdc/test-user.keytab.
Sep 25 07:34:17 dist-test-slave-57fn krb5kdc[29801](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321657, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T07:34:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T07:34:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790321392278483-9480-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790321392278483-9480-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 07:34:17.528891 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790321392278483-9480-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790321392278483-9480-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790321392278483-9480-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36383
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790321392278483-9480-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36383
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:34:17.607446 29814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:34:17.607666 29814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:34:17.607700 29814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:34:17.608783 29814 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:34:17.608843 29814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:34:17.608867 29814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:34:17.608886 29814 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:34:17.608954 29814 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:34:17.610409 29814 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790321392278483-9480-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790321392278483-9480-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36383
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790321392278483-9480-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36383
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29814
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790321392278483-9480-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:34:17.610642 29814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:34:17.610857 29814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:34:17.615262 29819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:34:17.615269 29820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:34:17.615358 29814 server_base.cc:1034] running on GCE node
W20260925 07:34:17.615269 29822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:34:17.615702 29814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:34:17.615998 29814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:34:17.617153 29814 hybrid_clock.cc:648] HybridClock initialized: now 1790321657617145 us; error 25 us; skew 500 ppm
Sep 25 07:34:17 dist-test-slave-57fn krb5kdc[29801](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321657, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:34:17.627785 29814 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 07:34:17.628216 29814 webserver.cc:460] Webserver started at http://127.0.0.1:44831/ using document root <none> and password file <none>
I20260925 07:34:17.628396 29814 fs_manager.cc:359] Metadata directory not provided
I20260925 07:34:17.628434 29814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:34:17.628517 29814 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:34:17.629469 29814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790321392278483-9480-0/minicluster-data/master-0/data/instance:
uuid: "22a4d984af0649c6adc3d6e51f09667a"
format_stamp: "Formatted at 2026-09-25 07:34:17 on dist-test-slave-57fn"
server_key: "7536a1ef054afb42db6812e990afca13"
server_key_iv: "eac35e5c19f49bb4fa3d80ccc90d96a5"
server_key_version: "encryptionkey@0"
I20260925 07:34:17.629849 29814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790321392278483-9480-0/minicluster-data/master-0/wal/instance:
uuid: "22a4d984af0649c6adc3d6e51f09667a"
format_stamp: "Formatted at 2026-09-25 07:34:17 on dist-test-slave-57fn"
server_key: "7536a1ef054afb42db6812e990afca13"
server_key_iv: "eac35e5c19f49bb4fa3d80ccc90d96a5"
server_key_version: "encryptionkey@0"
I20260925 07:34:17.631034 29814 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:34:17.631700 29829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:34:17.631862 29814 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 07:34:17.631958 29814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790321392278483-9480-0/minicluster-data/master-0/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790321392278483-9480-0/minicluster-data/master-0/wal
uuid: "22a4d984af0649c6adc3d6e51f09667a"
format_stamp: "Formatted at 2026-09-25 07:34:17 on dist-test-slave-57fn"
server_key: "7536a1ef054afb42db6812e990afca13"
server_key_iv: "eac35e5c19f49bb4fa3d80ccc90d96a5"
server_key_version: "encryptionkey@0"
I20260925 07:34:17.632030 29814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790321392278483-9480-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790321392278483-9480-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790321392278483-9480-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:34:17.656198 29814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:34:17.658022 29814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:34:17.658186 29814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:34:17.662082 29814 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36383
I20260925 07:34:17.662125 29881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36383 every 8 connection(s)
I20260925 07:34:17.662487 29814 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
I20260925 07:34:17.663425 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 29814
I20260925 07:34:17.663466 29882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:34:17.663544 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790321392278483-9480-0/minicluster-data/master-0/wal/instance
I20260925 07:34:17.663738 9480 external_mini_cluster.cc:1428] Setting key 5f1c8bc52f60d168f14238c3ba85e039
I20260925 07:34:17.667272 29882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22a4d984af0649c6adc3d6e51f09667a: Bootstrap starting.
I20260925 07:34:17.667908 29882 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 22a4d984af0649c6adc3d6e51f09667a: Neither blocks nor log segments found. Creating new log.
I20260925 07:34:17.668211 29882 log.cc:824] T 00000000000000000000000000000000 P 22a4d984af0649c6adc3d6e51f09667a: Log is configured to *not* fsync() on all Append() calls
I20260925 07:34:17.669051 29882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22a4d984af0649c6adc3d6e51f09667a: No bootstrap required, opened a new log
I20260925 07:34:17.670130 29885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:17.664268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46523 (local address 127.0.0.1:36383)
0925 07:34:17.664480 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:17.664489 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:17.664523 (+ 34us) server_negotiation.cc:407] Connection header received
0925 07:34:17.665205 (+ 682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:17.665214 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:17.665277 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:17.665435 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:34:17.665710 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:17.667044 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:17.667934 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:17.668562 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:17.668649 (+ 87us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:17.669399 (+ 750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:34:17.669403 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:34:17.669404 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 07:34:17.669649 (+ 245us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:34:17.669687 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 07:34:17.669701 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:34:17.669783 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:17.669877 (+ 94us) server_negotiation.cc:299] Negotiation successful
0925 07:34:17.669902 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":89,"threads_started":1}
I20260925 07:34:17.671268 9480 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 07:34:17.671584 29882 raft_consensus.cc:348] T 00000000000000000000000000000000 P 22a4d984af0649c6adc3d6e51f09667a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22a4d984af0649c6adc3d6e51f09667a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36383 } }
I20260925 07:34:17.671761 29882 raft_consensus.cc:374] T 00000000000000000000000000000000 P 22a4d984af0649c6adc3d6e51f09667a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:34:17.671808 29882 raft_consensus.cc:729] T 00000000000000000000000000000000 P 22a4d984af0649c6adc3d6e51f09667a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22a4d984af0649c6adc3d6e51f09667a, State: Initialized, Role: FOLLOWER
I20260925 07:34:17.671928 29882 consensus_queue.cc:260] T 00000000000000000000000000000000 P 22a4d984af0649c6adc3d6e51f09667a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22a4d984af0649c6adc3d6e51f09667a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36383 } }
I20260925 07:34:17.672042 29882 raft_consensus.cc:388] T 00000000000000000000000000000000 P 22a4d984af0649c6adc3d6e51f09667a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 07:34:17.672086 29882 raft_consensus.cc:482] T 00000000000000000000000000000000 P 22a4d984af0649c6adc3d6e51f09667a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 07:34:17.672133 29882 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 22a4d984af0649c6adc3d6e51f09667a [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:34:17.672673 29882 raft_consensus.cc:504] T 00000000000000000000000000000000 P 22a4d984af0649c6adc3d6e51f09667a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22a4d984af0649c6adc3d6e51f09667a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36383 } }
I20260925 07:34:17.672811 29882 leader_election.cc:302] T 00000000000000000000000000000000 P 22a4d984af0649c6adc3d6e51f09667a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 22a4d984af0649c6adc3d6e51f09667a; no voters:
I20260925 07:34:17.672979 29882 leader_election.cc:288] T 00000000000000000000000000000000 P 22a4d984af0649c6adc3d6e51f09667a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 07:34:17.673031 29892 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 22a4d984af0649c6adc3d6e51f09667a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:34:17.673187 29892 raft_consensus.cc:686] T 00000000000000000000000000000000 P 22a4d984af0649c6adc3d6e51f09667a [term 1 LEADER]: Becoming Leader. State: Replica: 22a4d984af0649c6adc3d6e51f09667a, State: Running, Role: LEADER
I20260925 07:34:17.673239 29882 sys_catalog.cc:564] T 00000000000000000000000000000000 P 22a4d984af0649c6adc3d6e51f09667a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:34:17.673282 29892 consensus_queue.cc:237] T 00000000000000000000000000000000 P 22a4d984af0649c6adc3d6e51f09667a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22a4d984af0649c6adc3d6e51f09667a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36383 } }
I20260925 07:34:17.673578 29892 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22a4d984af0649c6adc3d6e51f09667a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 22a4d984af0649c6adc3d6e51f09667a. Latest consensus state: current_term: 1 leader_uuid: "22a4d984af0649c6adc3d6e51f09667a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22a4d984af0649c6adc3d6e51f09667a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36383 } } }
I20260925 07:34:17.673695 29892 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22a4d984af0649c6adc3d6e51f09667a [sys.catalog]: This master's current role is: LEADER
I20260925 07:34:17.673588 29894 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22a4d984af0649c6adc3d6e51f09667a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "22a4d984af0649c6adc3d6e51f09667a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22a4d984af0649c6adc3d6e51f09667a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36383 } } }
I20260925 07:34:17.673810 29894 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22a4d984af0649c6adc3d6e51f09667a [sys.catalog]: This master's current role is: LEADER
I20260925 07:34:17.673887 29897 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 07:34:17.674350 29897 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 07:34:17.676494 29897 catalog_manager.cc:1347] Generated new cluster ID: 3e41a461f798469caebe9dc5dafda837
I20260925 07:34:17.676644 29897 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 07:34:17.677675 29885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:17.672764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45781 (local address 127.0.0.1:36383)
0925 07:34:17.672866 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:17.672867 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:17.672879 (+ 12us) server_negotiation.cc:407] Connection header received
0925 07:34:17.673057 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:17.673058 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:17.673070 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:17.673141 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:34:17.673364 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:17.674146 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:17.675172 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:17.675457 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:17.675540 (+ 83us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:17.676285 (+ 745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:34:17.676286 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:34:17.676287 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 07:34:17.676291 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:34:17.676304 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 07:34:17.676308 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:34:17.676381 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:17.677606 (+ 1225us) server_negotiation.cc:299] Negotiation successful
0925 07:34:17.677618 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 07:34:17.698392 29897 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 07:34:17.698922 29897 catalog_manager.cc:1504] Loading token signing keys...
I20260925 07:34:17.704566 29897 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 22a4d984af0649c6adc3d6e51f09667a: Generated new TSK 0
I20260925 07:34:17.704736 29897 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 07:34:17.707468 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 29814
W20260925 07:34:17.711818 29799 connection.cc:535] client connection to 127.0.0.1:36383 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36383): Connection reset by peer (error 104)
2026-09-25T07:34:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1320 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790321392278483-9480-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 07:34:17 dist-test-slave-57fn krb5kdc[29913](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 07:34:17 dist-test-slave-57fn krb5kdc[29913](info): set up 2 sockets
Sep 25 07:34:17 dist-test-slave-57fn krb5kdc[29913](info): commencing operation
krb5kdc: starting...
W20260925 07:34:18.749472 9480 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790321392278483-9480-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790321392278483-9480-0/krb5kdc/test-user.keytab.
Sep 25 07:34:18 dist-test-slave-57fn krb5kdc[29913](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321658, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T07:34:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T07:34:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790321392278483-9480-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790321392278483-9480-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 07:34:18.860745 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790321392278483-9480-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790321392278483-9480-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790321392278483-9480-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42979
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790321392278483-9480-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42979
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:34:18.936197 29926 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:34:18.936401 29926 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:34:18.936436 29926 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:34:18.937501 29926 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:34:18.937582 29926 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:34:18.937610 29926 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:34:18.937644 29926 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:34:18.937671 29926 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:34:18.939071 29926 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790321392278483-9480-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790321392278483-9480-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42979
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790321392278483-9480-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42979
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29926
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790321392278483-9480-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:34:18.939320 29926 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:34:18.939544 29926 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:34:18.944028 29931 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:34:18.944022 29934 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:34:18.944094 29932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:34:18.944172 29926 server_base.cc:1034] running on GCE node
I20260925 07:34:18.944466 29926 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:34:18.944730 29926 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:34:18.945885 29926 hybrid_clock.cc:648] HybridClock initialized: now 1790321658945866 us; error 28 us; skew 500 ppm
Sep 25 07:34:18 dist-test-slave-57fn krb5kdc[29913](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321658, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:34:18.948187 29926 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 07:34:18.948633 29926 webserver.cc:460] Webserver started at http://127.0.0.1:36855/ using document root <none> and password file <none>
I20260925 07:34:18.948832 29926 fs_manager.cc:359] Metadata directory not provided
I20260925 07:34:18.948904 29926 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:34:18.949038 29926 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:34:18.950006 29926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790321392278483-9480-0/minicluster-data/master-0/data/instance:
uuid: "8bbe7836449d4582b7b05e8e015b6faf"
format_stamp: "Formatted at 2026-09-25 07:34:18 on dist-test-slave-57fn"
server_key: "c810a80e699366ef08d0b07066489552"
server_key_iv: "f1fe769df9f9b532ebf5b36883659967"
server_key_version: "encryptionkey@0"
I20260925 07:34:18.950341 29926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790321392278483-9480-0/minicluster-data/master-0/wal/instance:
uuid: "8bbe7836449d4582b7b05e8e015b6faf"
format_stamp: "Formatted at 2026-09-25 07:34:18 on dist-test-slave-57fn"
server_key: "c810a80e699366ef08d0b07066489552"
server_key_iv: "f1fe769df9f9b532ebf5b36883659967"
server_key_version: "encryptionkey@0"
I20260925 07:34:18.951555 29926 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 07:34:18.952190 29941 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:34:18.952338 29926 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 07:34:18.952428 29926 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790321392278483-9480-0/minicluster-data/master-0/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790321392278483-9480-0/minicluster-data/master-0/wal
uuid: "8bbe7836449d4582b7b05e8e015b6faf"
format_stamp: "Formatted at 2026-09-25 07:34:18 on dist-test-slave-57fn"
server_key: "c810a80e699366ef08d0b07066489552"
server_key_iv: "f1fe769df9f9b532ebf5b36883659967"
server_key_version: "encryptionkey@0"
I20260925 07:34:18.952500 29926 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790321392278483-9480-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790321392278483-9480-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790321392278483-9480-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:34:18.955350 29926 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:34:18.956924 29926 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:34:18.957060 29926 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:34:18.961149 29926 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42979
I20260925 07:34:18.961184 29993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42979 every 8 connection(s)
I20260925 07:34:18.961480 29926 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
I20260925 07:34:18.962371 29994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:34:18.964255 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 29926
I20260925 07:34:18.964373 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790321392278483-9480-0/minicluster-data/master-0/wal/instance
I20260925 07:34:18.964538 9480 external_mini_cluster.cc:1428] Setting key e23a822443b94cc522fa9a5a4c62bf78
I20260925 07:34:18.964545 29994 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8bbe7836449d4582b7b05e8e015b6faf: Bootstrap starting.
I20260925 07:34:18.965260 29994 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8bbe7836449d4582b7b05e8e015b6faf: Neither blocks nor log segments found. Creating new log.
I20260925 07:34:18.965655 29994 log.cc:824] T 00000000000000000000000000000000 P 8bbe7836449d4582b7b05e8e015b6faf: Log is configured to *not* fsync() on all Append() calls
I20260925 07:34:18.966444 29994 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8bbe7836449d4582b7b05e8e015b6faf: No bootstrap required, opened a new log
I20260925 07:34:18.967090 29997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:18.965043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35169 (local address 127.0.0.1:42979)
0925 07:34:18.965262 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:18.965268 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:18.965290 (+ 22us) server_negotiation.cc:407] Connection header received
0925 07:34:18.965830 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:18.965852 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:18.965886 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:18.966029 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:34:18.966655 (+ 626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:34:18.966658 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:34:18.966658 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 07:34:18.966662 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:34:18.966692 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 07:34:18.966702 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:34:18.966767 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:18.966856 (+ 89us) server_negotiation.cc:299] Negotiation successful
0925 07:34:18.966884 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":108,"threads_started":1}
I20260925 07:34:18.968094 9480 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 07:34:18.968953 29994 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8bbe7836449d4582b7b05e8e015b6faf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bbe7836449d4582b7b05e8e015b6faf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42979 } }
I20260925 07:34:18.969117 29994 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8bbe7836449d4582b7b05e8e015b6faf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:34:18.969152 29994 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8bbe7836449d4582b7b05e8e015b6faf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bbe7836449d4582b7b05e8e015b6faf, State: Initialized, Role: FOLLOWER
I20260925 07:34:18.969273 29994 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8bbe7836449d4582b7b05e8e015b6faf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bbe7836449d4582b7b05e8e015b6faf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42979 } }
I20260925 07:34:18.969456 29994 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8bbe7836449d4582b7b05e8e015b6faf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 07:34:18.969532 29994 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8bbe7836449d4582b7b05e8e015b6faf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 07:34:18.969590 29994 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8bbe7836449d4582b7b05e8e015b6faf [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:34:18.970266 29994 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8bbe7836449d4582b7b05e8e015b6faf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bbe7836449d4582b7b05e8e015b6faf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42979 } }
I20260925 07:34:18.970394 29997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:18.969377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60945 (local address 127.0.0.1:42979)
0925 07:34:18.969410 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:18.969412 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:18.969492 (+ 80us) server_negotiation.cc:407] Connection header received
0925 07:34:18.969594 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:18.969595 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:18.969598 (+ 3us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:18.969643 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:34:18.970180 (+ 537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:34:18.970181 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:34:18.970182 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 07:34:18.970186 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:34:18.970196 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 07:34:18.970200 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:34:18.970251 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:18.970303 (+ 52us) server_negotiation.cc:299] Negotiation successful
0925 07:34:18.970316 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 07:34:18.970387 29994 leader_election.cc:302] T 00000000000000000000000000000000 P 8bbe7836449d4582b7b05e8e015b6faf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8bbe7836449d4582b7b05e8e015b6faf; no voters:
I20260925 07:34:18.970582 29994 leader_election.cc:288] T 00000000000000000000000000000000 P 8bbe7836449d4582b7b05e8e015b6faf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 07:34:18.970610 30004 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8bbe7836449d4582b7b05e8e015b6faf [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:34:18.970773 30004 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8bbe7836449d4582b7b05e8e015b6faf [term 1 LEADER]: Becoming Leader. State: Replica: 8bbe7836449d4582b7b05e8e015b6faf, State: Running, Role: LEADER
I20260925 07:34:18.970801 29994 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8bbe7836449d4582b7b05e8e015b6faf [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:34:18.970920 30004 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8bbe7836449d4582b7b05e8e015b6faf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bbe7836449d4582b7b05e8e015b6faf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42979 } }
I20260925 07:34:18.971315 30006 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bbe7836449d4582b7b05e8e015b6faf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8bbe7836449d4582b7b05e8e015b6faf. Latest consensus state: current_term: 1 leader_uuid: "8bbe7836449d4582b7b05e8e015b6faf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bbe7836449d4582b7b05e8e015b6faf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42979 } } }
I20260925 07:34:18.971341 30005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bbe7836449d4582b7b05e8e015b6faf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8bbe7836449d4582b7b05e8e015b6faf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bbe7836449d4582b7b05e8e015b6faf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42979 } } }
I20260925 07:34:18.971475 30005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bbe7836449d4582b7b05e8e015b6faf [sys.catalog]: This master's current role is: LEADER
I20260925 07:34:18.971683 30013 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 07:34:18.971874 30006 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bbe7836449d4582b7b05e8e015b6faf [sys.catalog]: This master's current role is: LEADER
I20260925 07:34:18.972163 30013 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 07:34:18.973660 30013 catalog_manager.cc:1347] Generated new cluster ID: 17b77fcdb7e94ea49fa4fdf7709a7d82
I20260925 07:34:18.973713 30013 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 07:34:19.020430 30013 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 07:34:19.020640 30013 catalog_manager.cc:1504] Loading token signing keys...
I20260925 07:34:19.030751 30013 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8bbe7836449d4582b7b05e8e015b6faf: Generated new TSK 0
I20260925 07:34:19.030956 30013 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 07:34:19.036039 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 29926
2026-09-25T07:34:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1328 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790321392278483-9480-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 07:34:19 dist-test-slave-57fn krb5kdc[30025](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 07:34:19 dist-test-slave-57fn krb5kdc[30025](info): set up 2 sockets
Sep 25 07:34:19 dist-test-slave-57fn krb5kdc[30025](info): commencing operation
krb5kdc: starting...
W20260925 07:34:20.065737 9480 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790321392278483-9480-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790321392278483-9480-0/krb5kdc/test-user.keytab.
Sep 25 07:34:20 dist-test-slave-57fn krb5kdc[30025](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321660, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T07:34:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T07:34:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790321392278483-9480-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790321392278483-9480-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 07:34:20.193279 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790321392278483-9480-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790321392278483-9480-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790321392278483-9480-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33971
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790321392278483-9480-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33971
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:34:20.271883 30038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:34:20.272079 30038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:34:20.272100 30038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:34:20.273120 30038 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:34:20.273152 30038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:34:20.273164 30038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:34:20.273176 30038 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:34:20.273186 30038 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:34:20.274549 30038 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790321392278483-9480-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790321392278483-9480-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33971
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790321392278483-9480-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33971
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30038
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790321392278483-9480-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:34:20.274727 30038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:34:20.274868 30038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:34:20.279326 30043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:34:20.279345 30044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:34:20.279464 30046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:34:20.279606 30038 server_base.cc:1034] running on GCE node
I20260925 07:34:20.279826 30038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:34:20.280128 30038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:34:20.281288 30038 hybrid_clock.cc:648] HybridClock initialized: now 1790321660281278 us; error 26 us; skew 500 ppm
Sep 25 07:34:20 dist-test-slave-57fn krb5kdc[30025](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321660, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:34:20.283658 30038 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 07:34:20.284122 30038 webserver.cc:460] Webserver started at http://127.0.0.1:37511/ using document root <none> and password file <none>
I20260925 07:34:20.284343 30038 fs_manager.cc:359] Metadata directory not provided
I20260925 07:34:20.284415 30038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:34:20.284535 30038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:34:20.285475 30038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790321392278483-9480-0/minicluster-data/master-0/data/instance:
uuid: "0f470cb82bfe47e79833d128346696a9"
format_stamp: "Formatted at 2026-09-25 07:34:20 on dist-test-slave-57fn"
server_key: "f1d937919a470e16b247bf08b2ea6c39"
server_key_iv: "bae2a359588c594ae0f25c66228d2849"
server_key_version: "encryptionkey@0"
I20260925 07:34:20.285894 30038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790321392278483-9480-0/minicluster-data/master-0/wal/instance:
uuid: "0f470cb82bfe47e79833d128346696a9"
format_stamp: "Formatted at 2026-09-25 07:34:20 on dist-test-slave-57fn"
server_key: "f1d937919a470e16b247bf08b2ea6c39"
server_key_iv: "bae2a359588c594ae0f25c66228d2849"
server_key_version: "encryptionkey@0"
I20260925 07:34:20.287089 30038 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:34:20.287828 30053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:34:20.288005 30038 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 07:34:20.288097 30038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790321392278483-9480-0/minicluster-data/master-0/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790321392278483-9480-0/minicluster-data/master-0/wal
uuid: "0f470cb82bfe47e79833d128346696a9"
format_stamp: "Formatted at 2026-09-25 07:34:20 on dist-test-slave-57fn"
server_key: "f1d937919a470e16b247bf08b2ea6c39"
server_key_iv: "bae2a359588c594ae0f25c66228d2849"
server_key_version: "encryptionkey@0"
I20260925 07:34:20.288180 30038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790321392278483-9480-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790321392278483-9480-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790321392278483-9480-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:34:20.295258 30038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:34:20.296854 30038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:34:20.297008 30038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:34:20.300877 30038 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33971
I20260925 07:34:20.300915 30105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33971 every 8 connection(s)
I20260925 07:34:20.301206 30038 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
I20260925 07:34:20.302148 30106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:34:20.304237 30106 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f470cb82bfe47e79833d128346696a9: Bootstrap starting.
I20260925 07:34:20.304800 30106 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0f470cb82bfe47e79833d128346696a9: Neither blocks nor log segments found. Creating new log.
I20260925 07:34:20.305075 30106 log.cc:824] T 00000000000000000000000000000000 P 0f470cb82bfe47e79833d128346696a9: Log is configured to *not* fsync() on all Append() calls
I20260925 07:34:20.305756 30106 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f470cb82bfe47e79833d128346696a9: No bootstrap required, opened a new log
I20260925 07:34:20.307248 30106 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0f470cb82bfe47e79833d128346696a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f470cb82bfe47e79833d128346696a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33971 } }
I20260925 07:34:20.307410 30106 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0f470cb82bfe47e79833d128346696a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:34:20.307435 30106 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0f470cb82bfe47e79833d128346696a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f470cb82bfe47e79833d128346696a9, State: Initialized, Role: FOLLOWER
I20260925 07:34:20.307480 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 30038
I20260925 07:34:20.307615 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790321392278483-9480-0/minicluster-data/master-0/wal/instance
I20260925 07:34:20.307590 30106 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0f470cb82bfe47e79833d128346696a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f470cb82bfe47e79833d128346696a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33971 } }
I20260925 07:34:20.307667 30106 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0f470cb82bfe47e79833d128346696a9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 07:34:20.307708 30106 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0f470cb82bfe47e79833d128346696a9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 07:34:20.307756 30106 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0f470cb82bfe47e79833d128346696a9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:34:20.307842 9480 external_mini_cluster.cc:1428] Setting key dbf31dbbb06d243c986d952298c04613
I20260925 07:34:20.308482 30106 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0f470cb82bfe47e79833d128346696a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f470cb82bfe47e79833d128346696a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33971 } }
I20260925 07:34:20.308607 30106 leader_election.cc:302] T 00000000000000000000000000000000 P 0f470cb82bfe47e79833d128346696a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0f470cb82bfe47e79833d128346696a9; no voters:
I20260925 07:34:20.308741 30106 leader_election.cc:288] T 00000000000000000000000000000000 P 0f470cb82bfe47e79833d128346696a9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 07:34:20.308815 30111 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0f470cb82bfe47e79833d128346696a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:34:20.308959 30111 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0f470cb82bfe47e79833d128346696a9 [term 1 LEADER]: Becoming Leader. State: Replica: 0f470cb82bfe47e79833d128346696a9, State: Running, Role: LEADER
I20260925 07:34:20.309036 30106 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0f470cb82bfe47e79833d128346696a9 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:34:20.309083 30111 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0f470cb82bfe47e79833d128346696a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f470cb82bfe47e79833d128346696a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33971 } }
I20260925 07:34:20.309361 30111 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f470cb82bfe47e79833d128346696a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f470cb82bfe47e79833d128346696a9. Latest consensus state: current_term: 1 leader_uuid: "0f470cb82bfe47e79833d128346696a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f470cb82bfe47e79833d128346696a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33971 } } }
I20260925 07:34:20.309448 30111 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f470cb82bfe47e79833d128346696a9 [sys.catalog]: This master's current role is: LEADER
I20260925 07:34:20.309372 30113 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f470cb82bfe47e79833d128346696a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0f470cb82bfe47e79833d128346696a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f470cb82bfe47e79833d128346696a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33971 } } }
I20260925 07:34:20.309614 30113 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f470cb82bfe47e79833d128346696a9 [sys.catalog]: This master's current role is: LEADER
I20260925 07:34:20.309660 30115 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 07:34:20.310225 30115 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 07:34:20.311928 30115 catalog_manager.cc:1347] Generated new cluster ID: e31e144be2824d0cb065e213bc55b6c7
I20260925 07:34:20.311981 30115 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 07:34:20 dist-test-slave-57fn krb5kdc[30025](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321660, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 07:34:20.319954 30110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:20.308375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57529 (local address 127.0.0.1:33971)
0925 07:34:20.308585 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:20.308593 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:20.309025 (+ 432us) server_negotiation.cc:407] Connection header received
0925 07:34:20.309133 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:20.309139 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:20.309200 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:20.309346 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:34:20.310442 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:20.311661 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:20.312726 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:20.313296 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:20.313359 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:20.316062 (+ 2703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:34:20.316065 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:34:20.316066 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:34:20.316077 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:34:20.318083 (+ 2006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:34:20.318472 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:34:20.318481 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:34:20.318481 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:34:20.318589 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:34:20.318781 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:34:20.318781 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:34:20.318782 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:34:20.319088 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:34:20.319225 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:20.319610 (+ 385us) server_negotiation.cc:299] Negotiation successful
0925 07:34:20.319734 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":93,"threads_started":1}
I20260925 07:34:20.320791 9480 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 07:34:20.324672 30115 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 07:34:20.325193 30115 catalog_manager.cc:1504] Loading token signing keys...
I20260925 07:34:20.331445 30115 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0f470cb82bfe47e79833d128346696a9: Generated new TSK 0
I20260925 07:34:20.331620 30115 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 07:34:20.332227 30110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:20.322208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60633 (local address 127.0.0.1:33971)
0925 07:34:20.322237 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:20.322239 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:20.324508 (+ 2269us) server_negotiation.cc:407] Connection header received
0925 07:34:20.324555 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:20.324556 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:20.324567 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:20.324619 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:34:20.326138 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:20.326857 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:20.327823 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:20.328148 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:20.328215 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:20.329461 (+ 1246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:34:20.329462 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:34:20.329463 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:34:20.329470 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:34:20.331058 (+ 1588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:34:20.331448 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:34:20.331449 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:34:20.331450 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:34:20.331531 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:34:20.331738 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:34:20.331739 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:34:20.331739 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:34:20.331865 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:34:20.331990 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:20.332061 (+ 71us) server_negotiation.cc:299] Negotiation successful
0925 07:34:20.332143 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260925 07:34:20.334034 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 30038
2026-09-25T07:34:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1298 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790321392278483-9480-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 07:34:20 dist-test-slave-57fn krb5kdc[30137](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 07:34:20 dist-test-slave-57fn krb5kdc[30137](info): set up 2 sockets
Sep 25 07:34:20 dist-test-slave-57fn krb5kdc[30137](info): commencing operation
krb5kdc: starting...
W20260925 07:34:21.345666 9480 mini_kdc.cc:121] Time spent starting KDC: real 0.994s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790321392278483-9480-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790321392278483-9480-0/krb5kdc/test-user.keytab.
Sep 25 07:34:21 dist-test-slave-57fn krb5kdc[30137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321661, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T07:34:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T07:34:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790321392278483-9480-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790321392278483-9480-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 07:34:21.462751 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790321392278483-9480-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790321392278483-9480-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790321392278483-9480-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33691
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790321392278483-9480-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33691
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:34:21.544315 30150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:34:21.544620 30150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:34:21.544646 30150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:34:21.545876 30150 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:34:21.545918 30150 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:34:21.545933 30150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:34:21.545945 30150 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:34:21.545956 30150 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:34:21.547477 30150 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790321392278483-9480-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790321392278483-9480-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33691
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790321392278483-9480-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33691
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30150
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790321392278483-9480-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:34:21.547716 30150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:34:21.547957 30150 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:34:21.552917 30155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:34:21.553015 30158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:34:21.553205 30150 server_base.cc:1034] running on GCE node
W20260925 07:34:21.553025 30156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:34:21.553496 30150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:34:21.553865 30150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:34:21.555085 30150 hybrid_clock.cc:648] HybridClock initialized: now 1790321661555072 us; error 32 us; skew 500 ppm
Sep 25 07:34:21 dist-test-slave-57fn krb5kdc[30137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321661, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:34:21.557482 30150 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 07:34:21.557976 30150 webserver.cc:460] Webserver started at http://127.0.0.1:44219/ using document root <none> and password file <none>
I20260925 07:34:21.558210 30150 fs_manager.cc:359] Metadata directory not provided
I20260925 07:34:21.558279 30150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:34:21.558429 30150 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:34:21.559428 30150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790321392278483-9480-0/minicluster-data/master-0/data/instance:
uuid: "ed5be774d5a7460bbef89b7b5638cf89"
format_stamp: "Formatted at 2026-09-25 07:34:21 on dist-test-slave-57fn"
server_key: "34a2a6f41d57e6e365d136155082eece"
server_key_iv: "98a5cdc26f9cc13d96fe79674cd858c9"
server_key_version: "encryptionkey@0"
I20260925 07:34:21.559782 30150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790321392278483-9480-0/minicluster-data/master-0/wal/instance:
uuid: "ed5be774d5a7460bbef89b7b5638cf89"
format_stamp: "Formatted at 2026-09-25 07:34:21 on dist-test-slave-57fn"
server_key: "34a2a6f41d57e6e365d136155082eece"
server_key_iv: "98a5cdc26f9cc13d96fe79674cd858c9"
server_key_version: "encryptionkey@0"
I20260925 07:34:21.561128 30150 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 07:34:21.561928 30165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:34:21.562107 30150 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 07:34:21.562201 30150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790321392278483-9480-0/minicluster-data/master-0/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790321392278483-9480-0/minicluster-data/master-0/wal
uuid: "ed5be774d5a7460bbef89b7b5638cf89"
format_stamp: "Formatted at 2026-09-25 07:34:21 on dist-test-slave-57fn"
server_key: "34a2a6f41d57e6e365d136155082eece"
server_key_iv: "98a5cdc26f9cc13d96fe79674cd858c9"
server_key_version: "encryptionkey@0"
I20260925 07:34:21.562288 30150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790321392278483-9480-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790321392278483-9480-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790321392278483-9480-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:34:21.591226 30150 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:34:21.593091 30150 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:34:21.593253 30150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:34:21.597342 30150 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33691
I20260925 07:34:21.597376 30217 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33691 every 8 connection(s)
I20260925 07:34:21.597761 30150 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
I20260925 07:34:21.598687 30218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:34:21.600903 30218 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed5be774d5a7460bbef89b7b5638cf89: Bootstrap starting.
I20260925 07:34:21.601451 30218 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ed5be774d5a7460bbef89b7b5638cf89: Neither blocks nor log segments found. Creating new log.
I20260925 07:34:21.601758 30218 log.cc:824] T 00000000000000000000000000000000 P ed5be774d5a7460bbef89b7b5638cf89: Log is configured to *not* fsync() on all Append() calls
I20260925 07:34:21.602459 30218 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed5be774d5a7460bbef89b7b5638cf89: No bootstrap required, opened a new log
I20260925 07:34:21.603992 30218 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed5be774d5a7460bbef89b7b5638cf89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed5be774d5a7460bbef89b7b5638cf89" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33691 } }
I20260925 07:34:21.604126 30218 raft_consensus.cc:374] T 00000000000000000000000000000000 P ed5be774d5a7460bbef89b7b5638cf89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:34:21.604149 30218 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed5be774d5a7460bbef89b7b5638cf89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed5be774d5a7460bbef89b7b5638cf89, State: Initialized, Role: FOLLOWER
I20260925 07:34:21.604339 30218 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed5be774d5a7460bbef89b7b5638cf89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed5be774d5a7460bbef89b7b5638cf89" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33691 } }
I20260925 07:34:21.604401 30218 raft_consensus.cc:388] T 00000000000000000000000000000000 P ed5be774d5a7460bbef89b7b5638cf89 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 07:34:21.604446 30218 raft_consensus.cc:482] T 00000000000000000000000000000000 P ed5be774d5a7460bbef89b7b5638cf89 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 07:34:21.604496 30218 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed5be774d5a7460bbef89b7b5638cf89 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:34:21.604993 30218 raft_consensus.cc:504] T 00000000000000000000000000000000 P ed5be774d5a7460bbef89b7b5638cf89 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed5be774d5a7460bbef89b7b5638cf89" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33691 } }
I20260925 07:34:21.605101 30218 leader_election.cc:302] T 00000000000000000000000000000000 P ed5be774d5a7460bbef89b7b5638cf89 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed5be774d5a7460bbef89b7b5638cf89; no voters:
I20260925 07:34:21.605276 30218 leader_election.cc:288] T 00000000000000000000000000000000 P ed5be774d5a7460bbef89b7b5638cf89 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 07:34:21.605295 30221 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ed5be774d5a7460bbef89b7b5638cf89 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:34:21.605481 30221 raft_consensus.cc:686] T 00000000000000000000000000000000 P ed5be774d5a7460bbef89b7b5638cf89 [term 1 LEADER]: Becoming Leader. State: Replica: ed5be774d5a7460bbef89b7b5638cf89, State: Running, Role: LEADER
I20260925 07:34:21.605552 30218 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed5be774d5a7460bbef89b7b5638cf89 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:34:21.605589 30221 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed5be774d5a7460bbef89b7b5638cf89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed5be774d5a7460bbef89b7b5638cf89" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33691 } }
I20260925 07:34:21.606000 30223 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed5be774d5a7460bbef89b7b5638cf89 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed5be774d5a7460bbef89b7b5638cf89. Latest consensus state: current_term: 1 leader_uuid: "ed5be774d5a7460bbef89b7b5638cf89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed5be774d5a7460bbef89b7b5638cf89" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33691 } } }
I20260925 07:34:21.606127 30223 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed5be774d5a7460bbef89b7b5638cf89 [sys.catalog]: This master's current role is: LEADER
I20260925 07:34:21.606117 30222 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed5be774d5a7460bbef89b7b5638cf89 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ed5be774d5a7460bbef89b7b5638cf89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed5be774d5a7460bbef89b7b5638cf89" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33691 } } }
I20260925 07:34:21.606511 30222 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed5be774d5a7460bbef89b7b5638cf89 [sys.catalog]: This master's current role is: LEADER
I20260925 07:34:21.606555 30230 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 07:34:21.607121 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 30150
I20260925 07:34:21.607225 30230 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 07:34:21.607237 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790321392278483-9480-0/minicluster-data/master-0/wal/instance
I20260925 07:34:21.607430 9480 external_mini_cluster.cc:1428] Setting key 1e888cde377dccc94ffb1c3f7aa8c4e4
I20260925 07:34:21.609097 30230 catalog_manager.cc:1347] Generated new cluster ID: 27039b5c1dba4176836484ec5e6ca66e
I20260925 07:34:21.609151 30230 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 07:34:21.614964 30240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:21.608014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46743 (local address 127.0.0.1:33691)
0925 07:34:21.609693 (+ 1679us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:21.609702 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:21.609733 (+ 31us) server_negotiation.cc:407] Connection header received
0925 07:34:21.609842 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:21.609848 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:21.609923 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:21.610069 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:34:21.610310 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:21.611753 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:21.612825 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:21.613463 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:21.613611 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:21.614308 (+ 697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:34:21.614312 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:34:21.614312 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 07:34:21.614509 (+ 197us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:34:21.614548 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 07:34:21.614562 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:34:21.614643 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:21.614697 (+ 54us) server_negotiation.cc:299] Negotiation successful
0925 07:34:21.614758 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1638,"thread_start_us":77,"threads_started":1}
I20260925 07:34:21.615808 9480 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 07:34:21.620622 30240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:21.617296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59951 (local address 127.0.0.1:33691)
0925 07:34:21.617327 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:21.617329 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:21.617408 (+ 79us) server_negotiation.cc:407] Connection header received
0925 07:34:21.617463 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:21.617463 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:21.617473 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:21.617553 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:34:21.617688 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:21.618400 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:21.619410 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:21.619693 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:21.619750 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:21.620391 (+ 641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:34:21.620392 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:34:21.620392 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 07:34:21.620396 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:34:21.620409 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 07:34:21.620413 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:34:21.620461 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:21.620533 (+ 72us) server_negotiation.cc:299] Negotiation successful
0925 07:34:21.620556 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 07:34:21.633432 30230 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 07:34:21.634171 30230 catalog_manager.cc:1504] Loading token signing keys...
I20260925 07:34:21.646045 30230 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ed5be774d5a7460bbef89b7b5638cf89: Generated new TSK 0
I20260925 07:34:21.646226 30230 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 07:34:21.651188 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 30150
2026-09-25T07:34:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1316 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790321392278483-9480-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 07:34:21 dist-test-slave-57fn krb5kdc[30249](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 07:34:21 dist-test-slave-57fn krb5kdc[30249](info): set up 2 sockets
Sep 25 07:34:21 dist-test-slave-57fn krb5kdc[30249](info): commencing operation
krb5kdc: starting...
W20260925 07:34:22.661816 9480 mini_kdc.cc:121] Time spent starting KDC: real 0.993s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790321392278483-9480-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790321392278483-9480-0/krb5kdc/test-user.keytab.
Sep 25 07:34:22 dist-test-slave-57fn krb5kdc[30249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321662, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T07:34:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T07:34:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790321392278483-9480-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790321392278483-9480-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 07:34:22.772914 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790321392278483-9480-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790321392278483-9480-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790321392278483-9480-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32983
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790321392278483-9480-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32983
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:34:22.850909 30262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:34:22.851130 30262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:34:22.851164 30262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:34:22.852231 30262 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:34:22.852298 30262 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:34:22.852320 30262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:34:22.852347 30262 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:34:22.852383 30262 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:34:22.853790 30262 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790321392278483-9480-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790321392278483-9480-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32983
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790321392278483-9480-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32983
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30262
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790321392278483-9480-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:34:22.854038 30262 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:34:22.854269 30262 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:34:22.858821 30267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:34:22.858832 30268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:34:22.858906 30262 server_base.cc:1034] running on GCE node
W20260925 07:34:22.858848 30270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:34:22.859257 30262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:34:22.859543 30262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:34:22.860707 30262 hybrid_clock.cc:648] HybridClock initialized: now 1790321662860694 us; error 27 us; skew 500 ppm
Sep 25 07:34:22 dist-test-slave-57fn krb5kdc[30249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321662, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:34:22.868860 30262 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 07:34:22.869282 30262 webserver.cc:460] Webserver started at http://127.0.0.1:41399/ using document root <none> and password file <none>
I20260925 07:34:22.869558 30262 fs_manager.cc:359] Metadata directory not provided
I20260925 07:34:22.869606 30262 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:34:22.869728 30262 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:34:22.870744 30262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790321392278483-9480-0/minicluster-data/master-0/data/instance:
uuid: "23ddf2ad8bf04769accbdb145bba86ff"
format_stamp: "Formatted at 2026-09-25 07:34:22 on dist-test-slave-57fn"
server_key: "0f4caaeee5aefdd7243d8f90e95541d8"
server_key_iv: "84ae2a0d94c9d2ea4c4d9484f012aad9"
server_key_version: "encryptionkey@0"
I20260925 07:34:22.871090 30262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790321392278483-9480-0/minicluster-data/master-0/wal/instance:
uuid: "23ddf2ad8bf04769accbdb145bba86ff"
format_stamp: "Formatted at 2026-09-25 07:34:22 on dist-test-slave-57fn"
server_key: "0f4caaeee5aefdd7243d8f90e95541d8"
server_key_iv: "84ae2a0d94c9d2ea4c4d9484f012aad9"
server_key_version: "encryptionkey@0"
I20260925 07:34:22.872349 30262 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 07:34:22.873060 30277 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:34:22.873267 30262 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:34:22.873358 30262 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790321392278483-9480-0/minicluster-data/master-0/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790321392278483-9480-0/minicluster-data/master-0/wal
uuid: "23ddf2ad8bf04769accbdb145bba86ff"
format_stamp: "Formatted at 2026-09-25 07:34:22 on dist-test-slave-57fn"
server_key: "0f4caaeee5aefdd7243d8f90e95541d8"
server_key_iv: "84ae2a0d94c9d2ea4c4d9484f012aad9"
server_key_version: "encryptionkey@0"
I20260925 07:34:22.873420 30262 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790321392278483-9480-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790321392278483-9480-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790321392278483-9480-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:34:22.876710 30262 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:34:22.878728 30262 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:34:22.878837 30262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:34:22.882558 30262 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32983
I20260925 07:34:22.882650 30329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32983 every 8 connection(s)
I20260925 07:34:22.882949 30262 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
I20260925 07:34:22.883816 30330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:34:22.885933 30330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23ddf2ad8bf04769accbdb145bba86ff: Bootstrap starting.
I20260925 07:34:22.886454 30330 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 23ddf2ad8bf04769accbdb145bba86ff: Neither blocks nor log segments found. Creating new log.
I20260925 07:34:22.886718 30330 log.cc:824] T 00000000000000000000000000000000 P 23ddf2ad8bf04769accbdb145bba86ff: Log is configured to *not* fsync() on all Append() calls
I20260925 07:34:22.886927 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 30262
I20260925 07:34:22.887061 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790321392278483-9480-0/minicluster-data/master-0/wal/instance
I20260925 07:34:22.887320 9480 external_mini_cluster.cc:1428] Setting key 256680c4cf84d7fd0e17a5bac37f6bf2
I20260925 07:34:22.887416 30330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23ddf2ad8bf04769accbdb145bba86ff: No bootstrap required, opened a new log
I20260925 07:34:22.889637 30330 raft_consensus.cc:348] T 00000000000000000000000000000000 P 23ddf2ad8bf04769accbdb145bba86ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23ddf2ad8bf04769accbdb145bba86ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32983 } }
I20260925 07:34:22.889799 30330 raft_consensus.cc:374] T 00000000000000000000000000000000 P 23ddf2ad8bf04769accbdb145bba86ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:34:22.889834 30330 raft_consensus.cc:729] T 00000000000000000000000000000000 P 23ddf2ad8bf04769accbdb145bba86ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23ddf2ad8bf04769accbdb145bba86ff, State: Initialized, Role: FOLLOWER
I20260925 07:34:22.889954 30330 consensus_queue.cc:260] T 00000000000000000000000000000000 P 23ddf2ad8bf04769accbdb145bba86ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23ddf2ad8bf04769accbdb145bba86ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32983 } }
I20260925 07:34:22.890062 30330 raft_consensus.cc:388] T 00000000000000000000000000000000 P 23ddf2ad8bf04769accbdb145bba86ff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 07:34:22.890110 30330 raft_consensus.cc:482] T 00000000000000000000000000000000 P 23ddf2ad8bf04769accbdb145bba86ff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 07:34:22.890118 30334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:22.887791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49243 (local address 127.0.0.1:32983)
0925 07:34:22.887986 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:22.887995 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:22.888027 (+ 32us) server_negotiation.cc:407] Connection header received
0925 07:34:22.888741 (+ 714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:22.888749 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:22.888782 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:22.888934 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:34:22.889590 (+ 656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:34:22.889593 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:34:22.889593 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 07:34:22.889597 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:34:22.889637 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 07:34:22.889651 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:34:22.889725 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:22.889774 (+ 49us) server_negotiation.cc:299] Negotiation successful
0925 07:34:22.889811 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":95,"threads_started":1}
I20260925 07:34:22.890161 30330 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 23ddf2ad8bf04769accbdb145bba86ff [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:34:22.890746 30330 raft_consensus.cc:504] T 00000000000000000000000000000000 P 23ddf2ad8bf04769accbdb145bba86ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23ddf2ad8bf04769accbdb145bba86ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32983 } }
I20260925 07:34:22.890863 30330 leader_election.cc:302] T 00000000000000000000000000000000 P 23ddf2ad8bf04769accbdb145bba86ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 23ddf2ad8bf04769accbdb145bba86ff; no voters:
I20260925 07:34:22.891079 30330 leader_election.cc:288] T 00000000000000000000000000000000 P 23ddf2ad8bf04769accbdb145bba86ff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 07:34:22.891127 30335 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 23ddf2ad8bf04769accbdb145bba86ff [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:34:22.891167 9480 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 07:34:22.891276 30335 raft_consensus.cc:686] T 00000000000000000000000000000000 P 23ddf2ad8bf04769accbdb145bba86ff [term 1 LEADER]: Becoming Leader. State: Replica: 23ddf2ad8bf04769accbdb145bba86ff, State: Running, Role: LEADER
I20260925 07:34:22.891278 30330 sys_catalog.cc:564] T 00000000000000000000000000000000 P 23ddf2ad8bf04769accbdb145bba86ff [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:34:22.891440 30335 consensus_queue.cc:237] T 00000000000000000000000000000000 P 23ddf2ad8bf04769accbdb145bba86ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23ddf2ad8bf04769accbdb145bba86ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32983 } }
I20260925 07:34:22.891791 30335 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23ddf2ad8bf04769accbdb145bba86ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 23ddf2ad8bf04769accbdb145bba86ff. Latest consensus state: current_term: 1 leader_uuid: "23ddf2ad8bf04769accbdb145bba86ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23ddf2ad8bf04769accbdb145bba86ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32983 } } }
I20260925 07:34:22.891891 30335 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23ddf2ad8bf04769accbdb145bba86ff [sys.catalog]: This master's current role is: LEADER
I20260925 07:34:22.891800 30336 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23ddf2ad8bf04769accbdb145bba86ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "23ddf2ad8bf04769accbdb145bba86ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23ddf2ad8bf04769accbdb145bba86ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32983 } } }
I20260925 07:34:22.892093 30336 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23ddf2ad8bf04769accbdb145bba86ff [sys.catalog]: This master's current role is: LEADER
W20260925 07:34:22.893677 30356 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 23ddf2ad8bf04769accbdb145bba86ff: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 07:34:22.893784 30356 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 07:34:22.893888 30334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:22.892726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46157 (local address 127.0.0.1:32983)
0925 07:34:22.892762 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:22.892764 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:22.892909 (+ 145us) server_negotiation.cc:407] Connection header received
0925 07:34:22.892984 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:22.892984 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:22.892989 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:22.893031 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:34:22.893631 (+ 600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:34:22.893632 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:34:22.893632 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 07:34:22.893637 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:34:22.893649 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 07:34:22.893654 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:34:22.893709 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:22.893768 (+ 59us) server_negotiation.cc:299] Negotiation successful
0925 07:34:22.893781 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 07:34:22.893908 30340 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 07:34:22.894073 30340 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 07:34:22.895478 30340 catalog_manager.cc:1347] Generated new cluster ID: 378175798966421ca90b47eb63c0a340
I20260925 07:34:22.895524 30340 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 07:34:22.905711 30340 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 07:34:22.905854 30340 catalog_manager.cc:1504] Loading token signing keys...
I20260925 07:34:22.918541 30340 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 23ddf2ad8bf04769accbdb145bba86ff: Generated new TSK 0
I20260925 07:34:22.918704 30340 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 07:34:22.923044 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 30262
2026-09-25T07:34:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1272 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790321392278483-9480-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 07:34:22 dist-test-slave-57fn krb5kdc[30361](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 07:34:22 dist-test-slave-57fn krb5kdc[30361](info): set up 2 sockets
Sep 25 07:34:22 dist-test-slave-57fn krb5kdc[30361](info): commencing operation
krb5kdc: starting...
W20260925 07:34:23.941473 9480 mini_kdc.cc:121] Time spent starting KDC: real 1.001s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790321392278483-9480-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790321392278483-9480-0/krb5kdc/test-user.keytab.
Sep 25 07:34:24 dist-test-slave-57fn krb5kdc[30361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321664, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T07:34:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T07:34:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790321392278483-9480-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790321392278483-9480-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 07:34:24.058274 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790321392278483-9480-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790321392278483-9480-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790321392278483-9480-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42119
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790321392278483-9480-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42119
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:34:24.136518 30374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:34:24.136759 30374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:34:24.136796 30374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:34:24.137974 30374 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:34:24.138046 30374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:34:24.138072 30374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:34:24.138105 30374 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:34:24.138130 30374 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:34:24.139575 30374 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790321392278483-9480-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790321392278483-9480-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42119
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790321392278483-9480-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42119
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30374
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790321392278483-9480-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:34:24.139842 30374 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:34:24.140100 30374 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:34:24.144798 30380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:34:24.144793 30382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:34:24.144793 30379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:34:24.144912 30374 server_base.cc:1034] running on GCE node
I20260925 07:34:24.145196 30374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:34:24.145483 30374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:34:24.146708 30374 hybrid_clock.cc:648] HybridClock initialized: now 1790321664146685 us; error 37 us; skew 500 ppm
Sep 25 07:34:24 dist-test-slave-57fn krb5kdc[30361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321664, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:34:24.149060 30374 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 07:34:24.149448 30374 webserver.cc:460] Webserver started at http://127.0.0.1:39709/ using document root <none> and password file <none>
I20260925 07:34:24.149708 30374 fs_manager.cc:359] Metadata directory not provided
I20260925 07:34:24.149777 30374 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:34:24.149922 30374 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:34:24.150897 30374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790321392278483-9480-0/minicluster-data/master-0/data/instance:
uuid: "adcc6826116f44afad18cd7895ddae37"
format_stamp: "Formatted at 2026-09-25 07:34:24 on dist-test-slave-57fn"
server_key: "02019c89d8917c7f5cfe04bb7611d48e"
server_key_iv: "dfbaf1bf92d95833372b8e4ff8c4bd6f"
server_key_version: "encryptionkey@0"
I20260925 07:34:24.151275 30374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790321392278483-9480-0/minicluster-data/master-0/wal/instance:
uuid: "adcc6826116f44afad18cd7895ddae37"
format_stamp: "Formatted at 2026-09-25 07:34:24 on dist-test-slave-57fn"
server_key: "02019c89d8917c7f5cfe04bb7611d48e"
server_key_iv: "dfbaf1bf92d95833372b8e4ff8c4bd6f"
server_key_version: "encryptionkey@0"
I20260925 07:34:24.152560 30374 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 07:34:24.153235 30389 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:34:24.153398 30374 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 07:34:24.153491 30374 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790321392278483-9480-0/minicluster-data/master-0/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790321392278483-9480-0/minicluster-data/master-0/wal
uuid: "adcc6826116f44afad18cd7895ddae37"
format_stamp: "Formatted at 2026-09-25 07:34:24 on dist-test-slave-57fn"
server_key: "02019c89d8917c7f5cfe04bb7611d48e"
server_key_iv: "dfbaf1bf92d95833372b8e4ff8c4bd6f"
server_key_version: "encryptionkey@0"
I20260925 07:34:24.153611 30374 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790321392278483-9480-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790321392278483-9480-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790321392278483-9480-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:34:24.161118 30374 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:34:24.162843 30374 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:34:24.162992 30374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:34:24.166699 30374 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42119
I20260925 07:34:24.166728 30441 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42119 every 8 connection(s)
I20260925 07:34:24.167053 30374 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
I20260925 07:34:24.167951 30442 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:34:24.170092 30442 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P adcc6826116f44afad18cd7895ddae37: Bootstrap starting.
I20260925 07:34:24.170634 30442 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P adcc6826116f44afad18cd7895ddae37: Neither blocks nor log segments found. Creating new log.
I20260925 07:34:24.170907 30442 log.cc:824] T 00000000000000000000000000000000 P adcc6826116f44afad18cd7895ddae37: Log is configured to *not* fsync() on all Append() calls
I20260925 07:34:24.171578 30442 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P adcc6826116f44afad18cd7895ddae37: No bootstrap required, opened a new log
I20260925 07:34:24.172343 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 30374
I20260925 07:34:24.172452 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790321392278483-9480-0/minicluster-data/master-0/wal/instance
I20260925 07:34:24.172655 9480 external_mini_cluster.cc:1428] Setting key 282bb6a3f2bb565576d42e915c3bfea4
I20260925 07:34:24.173906 30442 raft_consensus.cc:348] T 00000000000000000000000000000000 P adcc6826116f44afad18cd7895ddae37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adcc6826116f44afad18cd7895ddae37" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42119 } }
I20260925 07:34:24.174044 30442 raft_consensus.cc:374] T 00000000000000000000000000000000 P adcc6826116f44afad18cd7895ddae37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:34:24.174065 30442 raft_consensus.cc:729] T 00000000000000000000000000000000 P adcc6826116f44afad18cd7895ddae37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adcc6826116f44afad18cd7895ddae37, State: Initialized, Role: FOLLOWER
I20260925 07:34:24.174189 30442 consensus_queue.cc:260] T 00000000000000000000000000000000 P adcc6826116f44afad18cd7895ddae37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adcc6826116f44afad18cd7895ddae37" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42119 } }
I20260925 07:34:24.174247 30442 raft_consensus.cc:388] T 00000000000000000000000000000000 P adcc6826116f44afad18cd7895ddae37 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 07:34:24.174266 30442 raft_consensus.cc:482] T 00000000000000000000000000000000 P adcc6826116f44afad18cd7895ddae37 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 07:34:24.174350 30442 raft_consensus.cc:3037] T 00000000000000000000000000000000 P adcc6826116f44afad18cd7895ddae37 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:34:24.175043 30442 raft_consensus.cc:504] T 00000000000000000000000000000000 P adcc6826116f44afad18cd7895ddae37 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adcc6826116f44afad18cd7895ddae37" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42119 } }
I20260925 07:34:24.175158 30442 leader_election.cc:302] T 00000000000000000000000000000000 P adcc6826116f44afad18cd7895ddae37 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: adcc6826116f44afad18cd7895ddae37; no voters:
I20260925 07:34:24.175359 30442 leader_election.cc:288] T 00000000000000000000000000000000 P adcc6826116f44afad18cd7895ddae37 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 07:34:24.175403 30447 raft_consensus.cc:2781] T 00000000000000000000000000000000 P adcc6826116f44afad18cd7895ddae37 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:34:24.175576 30447 raft_consensus.cc:686] T 00000000000000000000000000000000 P adcc6826116f44afad18cd7895ddae37 [term 1 LEADER]: Becoming Leader. State: Replica: adcc6826116f44afad18cd7895ddae37, State: Running, Role: LEADER
I20260925 07:34:24.175609 30442 sys_catalog.cc:564] T 00000000000000000000000000000000 P adcc6826116f44afad18cd7895ddae37 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:34:24.175720 30447 consensus_queue.cc:237] T 00000000000000000000000000000000 P adcc6826116f44afad18cd7895ddae37 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adcc6826116f44afad18cd7895ddae37" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42119 } }
I20260925 07:34:24.176893 30449 sys_catalog.cc:455] T 00000000000000000000000000000000 P adcc6826116f44afad18cd7895ddae37 [sys.catalog]: SysCatalogTable state changed. Reason: New leader adcc6826116f44afad18cd7895ddae37. Latest consensus state: current_term: 1 leader_uuid: "adcc6826116f44afad18cd7895ddae37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adcc6826116f44afad18cd7895ddae37" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42119 } } }
I20260925 07:34:24.176986 30449 sys_catalog.cc:458] T 00000000000000000000000000000000 P adcc6826116f44afad18cd7895ddae37 [sys.catalog]: This master's current role is: LEADER
I20260925 07:34:24.176955 30448 sys_catalog.cc:455] T 00000000000000000000000000000000 P adcc6826116f44afad18cd7895ddae37 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "adcc6826116f44afad18cd7895ddae37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adcc6826116f44afad18cd7895ddae37" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42119 } } }
I20260925 07:34:24.177218 30448 sys_catalog.cc:458] T 00000000000000000000000000000000 P adcc6826116f44afad18cd7895ddae37 [sys.catalog]: This master's current role is: LEADER
W20260925 07:34:24.177383 30462 catalog_manager.cc:1558] T 00000000000000000000000000000000 P adcc6826116f44afad18cd7895ddae37: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 07:34:24.177457 30462 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 07:34:24.177536 30464 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 07:34:24.177686 30464 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 07:34:24.179109 30464 catalog_manager.cc:1347] Generated new cluster ID: 4cad5c51e4c548a49d6e154b6bf36d3c
I20260925 07:34:24.179162 30464 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 07:34:24 dist-test-slave-57fn krb5kdc[30361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321664, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 07:34:24.184953 30446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:24.173128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.118:43631 (local address 127.0.0.1:42119)
0925 07:34:24.173294 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:24.173302 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:24.173324 (+ 22us) server_negotiation.cc:407] Connection header received
0925 07:34:24.174092 (+ 768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:24.174102 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:24.174178 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:24.174353 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:34:24.175263 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:24.176705 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:24.177875 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:24.178466 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:24.178629 (+ 163us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:24.181455 (+ 2826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:34:24.181458 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:34:24.181459 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:34:24.181469 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:34:24.183336 (+ 1867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:34:24.183795 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:34:24.183797 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:34:24.183798 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:34:24.183894 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:34:24.184130 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:34:24.184131 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:34:24.184131 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:34:24.184392 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:34:24.184536 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:24.184679 (+ 143us) server_negotiation.cc:299] Negotiation successful
0925 07:34:24.184766 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":86,"threads_started":1}
I20260925 07:34:24.186105 9480 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 07:34:24.194239 30446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:24.187464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.118:53165 (local address 127.0.0.1:42119)
0925 07:34:24.187493 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:24.187495 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:24.187608 (+ 113us) server_negotiation.cc:407] Connection header received
0925 07:34:24.187683 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:24.187684 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:24.187692 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:24.187715 (+ 23us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:34:24.188324 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:24.189031 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:24.190006 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:24.190310 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:24.190344 (+ 34us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:24.191627 (+ 1283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:34:24.191628 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:34:24.191628 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:34:24.191634 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:34:24.193035 (+ 1401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:34:24.193402 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:34:24.193403 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:34:24.193403 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:34:24.193459 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:34:24.193711 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:34:24.193712 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:34:24.193712 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:34:24.193817 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:34:24.193881 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:24.194101 (+ 220us) server_negotiation.cc:299] Negotiation successful
0925 07:34:24.194157 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":14}
I20260925 07:34:24.195138 30464 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 07:34:24.195580 30464 catalog_manager.cc:1504] Loading token signing keys...
I20260925 07:34:24.204237 30464 catalog_manager.cc:5868] T 00000000000000000000000000000000 P adcc6826116f44afad18cd7895ddae37: Generated new TSK 0
I20260925 07:34:24.204397 30464 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 07:34:24.206494 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 30374
W20260925 07:34:24.210740 30359 connection.cc:535] client connection to 127.0.0.1:42119 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42119): Connection reset by peer (error 104)
2026-09-25T07:34:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1283 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790321392278483-9480-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 07:34:24 dist-test-slave-57fn krb5kdc[30473](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 07:34:24 dist-test-slave-57fn krb5kdc[30473](info): set up 2 sockets
Sep 25 07:34:24 dist-test-slave-57fn krb5kdc[30473](info): commencing operation
krb5kdc: starting...
W20260925 07:34:25.213379 9480 mini_kdc.cc:121] Time spent starting KDC: real 0.989s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790321392278483-9480-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790321392278483-9480-0/krb5kdc/test-user.keytab.
Sep 25 07:34:25 dist-test-slave-57fn krb5kdc[30473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321665, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T07:34:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T07:34:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790321392278483-9480-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790321392278483-9480-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 07:34:25.324627 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790321392278483-9480-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790321392278483-9480-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790321392278483-9480-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46523
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790321392278483-9480-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46523
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:34:25.400693 30486 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:34:25.400921 30486 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:34:25.400946 30486 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:34:25.402105 30486 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:34:25.402145 30486 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:34:25.402158 30486 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:34:25.402170 30486 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:34:25.402182 30486 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:34:25.403482 30486 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790321392278483-9480-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790321392278483-9480-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46523
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790321392278483-9480-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46523
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30486
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790321392278483-9480-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:34:25.403644 30486 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:34:25.403789 30486 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:34:25.408236 30494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:34:25.408264 30492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:34:25.408319 30486 server_base.cc:1034] running on GCE node
W20260925 07:34:25.408243 30491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:34:25.408689 30486 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:34:25.408960 30486 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:34:25.410099 30486 hybrid_clock.cc:648] HybridClock initialized: now 1790321665410081 us; error 34 us; skew 500 ppm
Sep 25 07:34:25 dist-test-slave-57fn krb5kdc[30473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321665, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:34:25.412509 30486 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 07:34:25.412848 30486 webserver.cc:460] Webserver started at http://127.0.0.1:40195/ using document root <none> and password file <none>
I20260925 07:34:25.413012 30486 fs_manager.cc:359] Metadata directory not provided
I20260925 07:34:25.413050 30486 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:34:25.413128 30486 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:34:25.414085 30486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790321392278483-9480-0/minicluster-data/master-0/data/instance:
uuid: "f8cb9964f02e45cd9b2e9a43df86cd03"
format_stamp: "Formatted at 2026-09-25 07:34:25 on dist-test-slave-57fn"
server_key: "e6c4f5f2ae099918d67bdac22531d55d"
server_key_iv: "74ba5ab8cdde9657c0daf749335987a3"
server_key_version: "encryptionkey@0"
I20260925 07:34:25.414358 30486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790321392278483-9480-0/minicluster-data/master-0/wal/instance:
uuid: "f8cb9964f02e45cd9b2e9a43df86cd03"
format_stamp: "Formatted at 2026-09-25 07:34:25 on dist-test-slave-57fn"
server_key: "e6c4f5f2ae099918d67bdac22531d55d"
server_key_iv: "74ba5ab8cdde9657c0daf749335987a3"
server_key_version: "encryptionkey@0"
I20260925 07:34:25.415503 30486 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 07:34:25.416224 30501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:34:25.416407 30486 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 07:34:25.416492 30486 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790321392278483-9480-0/minicluster-data/master-0/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790321392278483-9480-0/minicluster-data/master-0/wal
uuid: "f8cb9964f02e45cd9b2e9a43df86cd03"
format_stamp: "Formatted at 2026-09-25 07:34:25 on dist-test-slave-57fn"
server_key: "e6c4f5f2ae099918d67bdac22531d55d"
server_key_iv: "74ba5ab8cdde9657c0daf749335987a3"
server_key_version: "encryptionkey@0"
I20260925 07:34:25.416541 30486 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790321392278483-9480-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790321392278483-9480-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790321392278483-9480-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:34:25.428643 30486 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:34:25.430624 30486 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:34:25.430722 30486 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:34:25.434399 30486 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46523
I20260925 07:34:25.434456 30553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46523 every 8 connection(s)
I20260925 07:34:25.434757 30486 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
I20260925 07:34:25.435688 30554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:34:25.437879 30554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8cb9964f02e45cd9b2e9a43df86cd03: Bootstrap starting.
I20260925 07:34:25.438473 30554 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f8cb9964f02e45cd9b2e9a43df86cd03: Neither blocks nor log segments found. Creating new log.
I20260925 07:34:25.438478 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 30486
I20260925 07:34:25.438581 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790321392278483-9480-0/minicluster-data/master-0/wal/instance
I20260925 07:34:25.438771 9480 external_mini_cluster.cc:1428] Setting key cceedfd88423b332fc51f0e80f1bff77
I20260925 07:34:25.438807 30554 log.cc:824] T 00000000000000000000000000000000 P f8cb9964f02e45cd9b2e9a43df86cd03: Log is configured to *not* fsync() on all Append() calls
I20260925 07:34:25.439538 30554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8cb9964f02e45cd9b2e9a43df86cd03: No bootstrap required, opened a new log
I20260925 07:34:25.441391 30554 raft_consensus.cc:348] T 00000000000000000000000000000000 P f8cb9964f02e45cd9b2e9a43df86cd03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8cb9964f02e45cd9b2e9a43df86cd03" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46523 } }
I20260925 07:34:25.441555 30554 raft_consensus.cc:374] T 00000000000000000000000000000000 P f8cb9964f02e45cd9b2e9a43df86cd03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:34:25.441627 30554 raft_consensus.cc:729] T 00000000000000000000000000000000 P f8cb9964f02e45cd9b2e9a43df86cd03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8cb9964f02e45cd9b2e9a43df86cd03, State: Initialized, Role: FOLLOWER
I20260925 07:34:25.441766 30554 consensus_queue.cc:260] T 00000000000000000000000000000000 P f8cb9964f02e45cd9b2e9a43df86cd03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8cb9964f02e45cd9b2e9a43df86cd03" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46523 } }
I20260925 07:34:25.441843 30554 raft_consensus.cc:388] T 00000000000000000000000000000000 P f8cb9964f02e45cd9b2e9a43df86cd03 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 07:34:25.441884 30554 raft_consensus.cc:482] T 00000000000000000000000000000000 P f8cb9964f02e45cd9b2e9a43df86cd03 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 07:34:25.441929 30554 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f8cb9964f02e45cd9b2e9a43df86cd03 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:34:25.442411 30554 raft_consensus.cc:504] T 00000000000000000000000000000000 P f8cb9964f02e45cd9b2e9a43df86cd03 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8cb9964f02e45cd9b2e9a43df86cd03" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46523 } }
I20260925 07:34:25.442523 30554 leader_election.cc:302] T 00000000000000000000000000000000 P f8cb9964f02e45cd9b2e9a43df86cd03 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f8cb9964f02e45cd9b2e9a43df86cd03; no voters:
I20260925 07:34:25.442687 30554 leader_election.cc:288] T 00000000000000000000000000000000 P f8cb9964f02e45cd9b2e9a43df86cd03 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 07:34:25.442735 30559 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f8cb9964f02e45cd9b2e9a43df86cd03 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:34:25.442909 30554 sys_catalog.cc:564] T 00000000000000000000000000000000 P f8cb9964f02e45cd9b2e9a43df86cd03 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:34:25.442945 30559 raft_consensus.cc:686] T 00000000000000000000000000000000 P f8cb9964f02e45cd9b2e9a43df86cd03 [term 1 LEADER]: Becoming Leader. State: Replica: f8cb9964f02e45cd9b2e9a43df86cd03, State: Running, Role: LEADER
I20260925 07:34:25.443395 30559 consensus_queue.cc:237] T 00000000000000000000000000000000 P f8cb9964f02e45cd9b2e9a43df86cd03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8cb9964f02e45cd9b2e9a43df86cd03" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46523 } }
I20260925 07:34:25.443954 30560 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8cb9964f02e45cd9b2e9a43df86cd03 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f8cb9964f02e45cd9b2e9a43df86cd03" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8cb9964f02e45cd9b2e9a43df86cd03" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46523 } } }
I20260925 07:34:25.444038 30560 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8cb9964f02e45cd9b2e9a43df86cd03 [sys.catalog]: This master's current role is: LEADER
I20260925 07:34:25.444375 30558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:25.439241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.118:58939 (local address 127.0.0.1:46523)
0925 07:34:25.439430 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:25.439438 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:25.439473 (+ 35us) server_negotiation.cc:407] Connection header received
0925 07:34:25.439928 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:25.439935 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:25.439981 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:25.440112 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:34:25.440331 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:25.441168 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:25.442206 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:25.442693 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:25.443119 (+ 426us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:25.443542 (+ 423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:34:25.443546 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:34:25.443547 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 07:34:25.443755 (+ 208us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:34:25.443788 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 07:34:25.443802 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:34:25.444074 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:25.444125 (+ 51us) server_negotiation.cc:299] Negotiation successful
0925 07:34:25.444155 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":77,"threads_started":1}
I20260925 07:34:25.444340 30561 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8cb9964f02e45cd9b2e9a43df86cd03 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f8cb9964f02e45cd9b2e9a43df86cd03. Latest consensus state: current_term: 1 leader_uuid: "f8cb9964f02e45cd9b2e9a43df86cd03" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8cb9964f02e45cd9b2e9a43df86cd03" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46523 } } }
I20260925 07:34:25.444429 30571 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 07:34:25.444718 30561 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8cb9964f02e45cd9b2e9a43df86cd03 [sys.catalog]: This master's current role is: LEADER
I20260925 07:34:25.445053 30571 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 07:34:25.446579 9480 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 07:34:25.449873 30571 catalog_manager.cc:1347] Generated new cluster ID: 1c1f9702a072472c88085e026d644b84
I20260925 07:34:25.449921 30571 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 07:34:25.454907 30558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:25.447908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.118:60467 (local address 127.0.0.1:46523)
0925 07:34:25.447945 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:25.447947 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:25.448031 (+ 84us) server_negotiation.cc:407] Connection header received
0925 07:34:25.448087 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:25.448088 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:25.448095 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:25.448135 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:34:25.448261 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:25.448690 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:25.449693 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:25.453834 (+ 4141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:25.453898 (+ 64us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:25.454567 (+ 669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:34:25.454568 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:34:25.454569 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 07:34:25.454573 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:34:25.454585 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 07:34:25.454589 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:34:25.454644 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:25.454834 (+ 190us) server_negotiation.cc:299] Negotiation successful
0925 07:34:25.454843 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260925 07:34:25.474572 30571 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 07:34:25.475102 30571 catalog_manager.cc:1504] Loading token signing keys...
I20260925 07:34:25.486526 30571 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f8cb9964f02e45cd9b2e9a43df86cd03: Generated new TSK 0
I20260925 07:34:25.486728 30571 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 07:34:25.497650 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 30486
W20260925 07:34:25.501907 30471 connection.cc:535] client connection to 127.0.0.1:46523 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46523): Connection reset by peer (error 104)
2026-09-25T07:34:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1291 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790321392278483-9480-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 07:34:25 dist-test-slave-57fn krb5kdc[30585](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 07:34:25 dist-test-slave-57fn krb5kdc[30585](info): set up 2 sockets
Sep 25 07:34:25 dist-test-slave-57fn krb5kdc[30585](info): commencing operation
krb5kdc: starting...
W20260925 07:34:26.517455 9480 mini_kdc.cc:121] Time spent starting KDC: real 1.002s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790321392278483-9480-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790321392278483-9480-0/krb5kdc/test-user.keytab.
Sep 25 07:34:26 dist-test-slave-57fn krb5kdc[30585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321666, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T07:34:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T07:34:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790321392278483-9480-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790321392278483-9480-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 07:34:26.633106 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790321392278483-9480-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790321392278483-9480-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790321392278483-9480-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46079
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790321392278483-9480-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46079
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:34:26.711324 30598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:34:26.711556 30598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:34:26.711596 30598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:34:26.712713 30598 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:34:26.712779 30598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:34:26.712800 30598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:34:26.712822 30598 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:34:26.712853 30598 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:34:26.714323 30598 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790321392278483-9480-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790321392278483-9480-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46079
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790321392278483-9480-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46079
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30598
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790321392278483-9480-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:34:26.714591 30598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:34:26.714795 30598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:34:26.719393 30604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:34:26.719401 30606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:34:26.719401 30603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:34:26.719684 30598 server_base.cc:1034] running on GCE node
I20260925 07:34:26.719918 30598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:34:26.720211 30598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:34:26.721362 30598 hybrid_clock.cc:648] HybridClock initialized: now 1790321666721342 us; error 35 us; skew 500 ppm
Sep 25 07:34:26 dist-test-slave-57fn krb5kdc[30585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321666, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:34:26.723812 30598 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 07:34:26.724249 30598 webserver.cc:460] Webserver started at http://127.0.0.1:36127/ using document root <none> and password file <none>
I20260925 07:34:26.724498 30598 fs_manager.cc:359] Metadata directory not provided
I20260925 07:34:26.724571 30598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:34:26.724715 30598 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:34:26.725780 30598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790321392278483-9480-0/minicluster-data/master-0/data/instance:
uuid: "01c3415d3b2048b290fc65345f6bc067"
format_stamp: "Formatted at 2026-09-25 07:34:26 on dist-test-slave-57fn"
server_key: "65b318ebed118211a6c1952f06ea5af1"
server_key_iv: "b1e3048596b55fd04f2a2aea761190f0"
server_key_version: "encryptionkey@0"
I20260925 07:34:26.726121 30598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790321392278483-9480-0/minicluster-data/master-0/wal/instance:
uuid: "01c3415d3b2048b290fc65345f6bc067"
format_stamp: "Formatted at 2026-09-25 07:34:26 on dist-test-slave-57fn"
server_key: "65b318ebed118211a6c1952f06ea5af1"
server_key_iv: "b1e3048596b55fd04f2a2aea761190f0"
server_key_version: "encryptionkey@0"
I20260925 07:34:26.727349 30598 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:34:26.728039 30613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:34:26.728199 30598 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 07:34:26.728314 30598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790321392278483-9480-0/minicluster-data/master-0/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790321392278483-9480-0/minicluster-data/master-0/wal
uuid: "01c3415d3b2048b290fc65345f6bc067"
format_stamp: "Formatted at 2026-09-25 07:34:26 on dist-test-slave-57fn"
server_key: "65b318ebed118211a6c1952f06ea5af1"
server_key_iv: "b1e3048596b55fd04f2a2aea761190f0"
server_key_version: "encryptionkey@0"
I20260925 07:34:26.728385 30598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790321392278483-9480-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790321392278483-9480-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790321392278483-9480-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:34:26.731671 30598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:34:26.733929 30598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:34:26.734086 30598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:34:26.738085 30598 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46079
I20260925 07:34:26.738114 30665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46079 every 8 connection(s)
I20260925 07:34:26.738426 30598 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
I20260925 07:34:26.739326 30666 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:34:26.741359 30666 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01c3415d3b2048b290fc65345f6bc067: Bootstrap starting.
I20260925 07:34:26.741945 30666 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 01c3415d3b2048b290fc65345f6bc067: Neither blocks nor log segments found. Creating new log.
I20260925 07:34:26.742256 30666 log.cc:824] T 00000000000000000000000000000000 P 01c3415d3b2048b290fc65345f6bc067: Log is configured to *not* fsync() on all Append() calls
I20260925 07:34:26.743083 30666 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01c3415d3b2048b290fc65345f6bc067: No bootstrap required, opened a new log
I20260925 07:34:26.744686 30666 raft_consensus.cc:348] T 00000000000000000000000000000000 P 01c3415d3b2048b290fc65345f6bc067 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01c3415d3b2048b290fc65345f6bc067" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46079 } }
I20260925 07:34:26.744812 30666 raft_consensus.cc:374] T 00000000000000000000000000000000 P 01c3415d3b2048b290fc65345f6bc067 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:34:26.744834 30666 raft_consensus.cc:729] T 00000000000000000000000000000000 P 01c3415d3b2048b290fc65345f6bc067 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01c3415d3b2048b290fc65345f6bc067, State: Initialized, Role: FOLLOWER
I20260925 07:34:26.745000 30666 consensus_queue.cc:260] T 00000000000000000000000000000000 P 01c3415d3b2048b290fc65345f6bc067 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01c3415d3b2048b290fc65345f6bc067" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46079 } }
I20260925 07:34:26.745064 30666 raft_consensus.cc:388] T 00000000000000000000000000000000 P 01c3415d3b2048b290fc65345f6bc067 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 07:34:26.745105 30666 raft_consensus.cc:482] T 00000000000000000000000000000000 P 01c3415d3b2048b290fc65345f6bc067 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 07:34:26.745152 30666 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 01c3415d3b2048b290fc65345f6bc067 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:34:26.745759 30666 raft_consensus.cc:504] T 00000000000000000000000000000000 P 01c3415d3b2048b290fc65345f6bc067 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01c3415d3b2048b290fc65345f6bc067" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46079 } }
I20260925 07:34:26.745842 30666 leader_election.cc:302] T 00000000000000000000000000000000 P 01c3415d3b2048b290fc65345f6bc067 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 01c3415d3b2048b290fc65345f6bc067; no voters:
I20260925 07:34:26.746016 30666 leader_election.cc:288] T 00000000000000000000000000000000 P 01c3415d3b2048b290fc65345f6bc067 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 07:34:26.746060 30669 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 01c3415d3b2048b290fc65345f6bc067 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:34:26.746233 30669 raft_consensus.cc:686] T 00000000000000000000000000000000 P 01c3415d3b2048b290fc65345f6bc067 [term 1 LEADER]: Becoming Leader. State: Replica: 01c3415d3b2048b290fc65345f6bc067, State: Running, Role: LEADER
I20260925 07:34:26.746276 30666 sys_catalog.cc:564] T 00000000000000000000000000000000 P 01c3415d3b2048b290fc65345f6bc067 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:34:26.746343 30669 consensus_queue.cc:237] T 00000000000000000000000000000000 P 01c3415d3b2048b290fc65345f6bc067 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01c3415d3b2048b290fc65345f6bc067" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46079 } }
I20260925 07:34:26.746690 30671 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01c3415d3b2048b290fc65345f6bc067 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 01c3415d3b2048b290fc65345f6bc067. Latest consensus state: current_term: 1 leader_uuid: "01c3415d3b2048b290fc65345f6bc067" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01c3415d3b2048b290fc65345f6bc067" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46079 } } }
I20260925 07:34:26.746819 30671 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01c3415d3b2048b290fc65345f6bc067 [sys.catalog]: This master's current role is: LEADER
I20260925 07:34:26.746815 30670 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01c3415d3b2048b290fc65345f6bc067 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "01c3415d3b2048b290fc65345f6bc067" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01c3415d3b2048b290fc65345f6bc067" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46079 } } }
I20260925 07:34:26.747169 30670 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01c3415d3b2048b290fc65345f6bc067 [sys.catalog]: This master's current role is: LEADER
I20260925 07:34:26.747216 30678 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 07:34:26.747278 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 30598
I20260925 07:34:26.747390 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790321392278483-9480-0/minicluster-data/master-0/wal/instance
I20260925 07:34:26.747596 9480 external_mini_cluster.cc:1428] Setting key 4f9932c1c73ba83b8cebbf052cc070db
I20260925 07:34:26.748059 30678 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 07:34:26.749773 30678 catalog_manager.cc:1347] Generated new cluster ID: 2b712b46971447e39a457fd3e6da15f6
I20260925 07:34:26.749826 30678 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 07:34:26.750108 30682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:26.748039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.118:45801 (local address 127.0.0.1:46079)
0925 07:34:26.748218 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:26.748227 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:26.748258 (+ 31us) server_negotiation.cc:407] Connection header received
0925 07:34:26.748744 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:26.748752 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:26.748787 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:26.748933 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:34:26.749594 (+ 661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:34:26.749597 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:34:26.749597 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 07:34:26.749601 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:34:26.749643 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 07:34:26.749658 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:34:26.749862 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:26.749892 (+ 30us) server_negotiation.cc:299] Negotiation successful
0925 07:34:26.749942 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":101,"threads_started":1}
I20260925 07:34:26.751026 9480 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 07:34:26.753362 30682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:26.752404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.118:38147 (local address 127.0.0.1:46079)
0925 07:34:26.752435 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:26.752436 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:26.752535 (+ 99us) server_negotiation.cc:407] Connection header received
0925 07:34:26.752575 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:26.752576 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:26.752580 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:26.752627 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:34:26.753158 (+ 531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:34:26.753159 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:34:26.753159 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 07:34:26.753163 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:34:26.753177 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 07:34:26.753181 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:34:26.753242 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:26.753292 (+ 50us) server_negotiation.cc:299] Negotiation successful
0925 07:34:26.753304 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 07:34:26.756583 30678 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 07:34:26.756732 30678 catalog_manager.cc:1504] Loading token signing keys...
I20260925 07:34:26.763664 30678 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 01c3415d3b2048b290fc65345f6bc067: Generated new TSK 0
I20260925 07:34:26.763823 30678 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 07:34:26.770098 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 30598
2026-09-25T07:34:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1272 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790321392278483-9480-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 07:34:26 dist-test-slave-57fn krb5kdc[30697](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 07:34:26 dist-test-slave-57fn krb5kdc[30697](info): set up 2 sockets
Sep 25 07:34:26 dist-test-slave-57fn krb5kdc[30697](info): commencing operation
krb5kdc: starting...
W20260925 07:34:27.809650 9480 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790321392278483-9480-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790321392278483-9480-0/krb5kdc/test-user.keytab.
Sep 25 07:34:27 dist-test-slave-57fn krb5kdc[30697](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321667, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T07:34:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T07:34:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790321392278483-9480-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790321392278483-9480-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 07:34:27.921124 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790321392278483-9480-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790321392278483-9480-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790321392278483-9480-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40639
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790321392278483-9480-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40639
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:34:27.997869 30710 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:34:27.998082 30710 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:34:27.998116 30710 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:34:27.999253 30710 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:34:27.999318 30710 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:34:27.999342 30710 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:34:27.999364 30710 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:34:27.999392 30710 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:34:28.000833 30710 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790321392278483-9480-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790321392278483-9480-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40639
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790321392278483-9480-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40639
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30710
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790321392278483-9480-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:34:28.001094 30710 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:34:28.001325 30710 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:34:28.005914 30715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:34:28.005966 30716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:34:28.005968 30718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:34:28.006017 30710 server_base.cc:1034] running on GCE node
I20260925 07:34:28.006331 30710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:34:28.006657 30710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:34:28.007817 30710 hybrid_clock.cc:648] HybridClock initialized: now 1790321668007817 us; error 41 us; skew 500 ppm
Sep 25 07:34:28 dist-test-slave-57fn krb5kdc[30697](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321668, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:34:28.017035 30710 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 07:34:28.017482 30710 webserver.cc:460] Webserver started at http://127.0.0.1:44089/ using document root <none> and password file <none>
I20260925 07:34:28.017733 30710 fs_manager.cc:359] Metadata directory not provided
I20260925 07:34:28.017807 30710 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:34:28.017951 30710 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:34:28.018990 30710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790321392278483-9480-0/minicluster-data/master-0/data/instance:
uuid: "525ffb414e524020ab1bff4ba9ec806f"
format_stamp: "Formatted at 2026-09-25 07:34:28 on dist-test-slave-57fn"
server_key: "bb85a4c7aed509efa069a7ae7ca59d96"
server_key_iv: "726561ebf6fda5fa47360682401b7470"
server_key_version: "encryptionkey@0"
I20260925 07:34:28.019349 30710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790321392278483-9480-0/minicluster-data/master-0/wal/instance:
uuid: "525ffb414e524020ab1bff4ba9ec806f"
format_stamp: "Formatted at 2026-09-25 07:34:28 on dist-test-slave-57fn"
server_key: "bb85a4c7aed509efa069a7ae7ca59d96"
server_key_iv: "726561ebf6fda5fa47360682401b7470"
server_key_version: "encryptionkey@0"
I20260925 07:34:28.020638 30710 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 07:34:28.021322 30725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:34:28.021497 30710 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 07:34:28.021641 30710 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790321392278483-9480-0/minicluster-data/master-0/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790321392278483-9480-0/minicluster-data/master-0/wal
uuid: "525ffb414e524020ab1bff4ba9ec806f"
format_stamp: "Formatted at 2026-09-25 07:34:28 on dist-test-slave-57fn"
server_key: "bb85a4c7aed509efa069a7ae7ca59d96"
server_key_iv: "726561ebf6fda5fa47360682401b7470"
server_key_version: "encryptionkey@0"
I20260925 07:34:28.021737 30710 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790321392278483-9480-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790321392278483-9480-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790321392278483-9480-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:34:28.057615 30710 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:34:28.059643 30710 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:34:28.059801 30710 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:34:28.063911 30710 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40639
I20260925 07:34:28.063953 30777 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40639 every 8 connection(s)
I20260925 07:34:28.064285 30710 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
I20260925 07:34:28.065163 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 30710
I20260925 07:34:28.065279 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790321392278483-9480-0/minicluster-data/master-0/wal/instance
I20260925 07:34:28.065280 30778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:34:28.065500 9480 external_mini_cluster.cc:1428] Setting key 91af8eed84ff23c58a438d84568fb7bc
I20260925 07:34:28.069038 30778 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 525ffb414e524020ab1bff4ba9ec806f: Bootstrap starting.
I20260925 07:34:28.069772 30778 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 525ffb414e524020ab1bff4ba9ec806f: Neither blocks nor log segments found. Creating new log.
I20260925 07:34:28.070075 30778 log.cc:824] T 00000000000000000000000000000000 P 525ffb414e524020ab1bff4ba9ec806f: Log is configured to *not* fsync() on all Append() calls
I20260925 07:34:28.070847 30778 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 525ffb414e524020ab1bff4ba9ec806f: No bootstrap required, opened a new log
Sep 25 07:34:28 dist-test-slave-57fn krb5kdc[30697](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321667, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 07:34:28.073050 30778 raft_consensus.cc:348] T 00000000000000000000000000000000 P 525ffb414e524020ab1bff4ba9ec806f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "525ffb414e524020ab1bff4ba9ec806f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40639 } }
I20260925 07:34:28.073206 30778 raft_consensus.cc:374] T 00000000000000000000000000000000 P 525ffb414e524020ab1bff4ba9ec806f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:34:28.073253 30778 raft_consensus.cc:729] T 00000000000000000000000000000000 P 525ffb414e524020ab1bff4ba9ec806f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525ffb414e524020ab1bff4ba9ec806f, State: Initialized, Role: FOLLOWER
I20260925 07:34:28.073405 30778 consensus_queue.cc:260] T 00000000000000000000000000000000 P 525ffb414e524020ab1bff4ba9ec806f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "525ffb414e524020ab1bff4ba9ec806f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40639 } }
I20260925 07:34:28.073521 30778 raft_consensus.cc:388] T 00000000000000000000000000000000 P 525ffb414e524020ab1bff4ba9ec806f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 07:34:28.073562 30778 raft_consensus.cc:482] T 00000000000000000000000000000000 P 525ffb414e524020ab1bff4ba9ec806f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 07:34:28.073623 30778 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 525ffb414e524020ab1bff4ba9ec806f [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:34:28.074370 30778 raft_consensus.cc:504] T 00000000000000000000000000000000 P 525ffb414e524020ab1bff4ba9ec806f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "525ffb414e524020ab1bff4ba9ec806f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40639 } }
I20260925 07:34:28.074489 30778 leader_election.cc:302] T 00000000000000000000000000000000 P 525ffb414e524020ab1bff4ba9ec806f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 525ffb414e524020ab1bff4ba9ec806f; no voters:
I20260925 07:34:28.074653 30778 leader_election.cc:288] T 00000000000000000000000000000000 P 525ffb414e524020ab1bff4ba9ec806f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 07:34:28.074707 30783 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 525ffb414e524020ab1bff4ba9ec806f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:34:28.074865 30783 raft_consensus.cc:686] T 00000000000000000000000000000000 P 525ffb414e524020ab1bff4ba9ec806f [term 1 LEADER]: Becoming Leader. State: Replica: 525ffb414e524020ab1bff4ba9ec806f, State: Running, Role: LEADER
I20260925 07:34:28.074914 30778 sys_catalog.cc:564] T 00000000000000000000000000000000 P 525ffb414e524020ab1bff4ba9ec806f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:34:28.075060 30783 consensus_queue.cc:237] T 00000000000000000000000000000000 P 525ffb414e524020ab1bff4ba9ec806f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "525ffb414e524020ab1bff4ba9ec806f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40639 } }
I20260925 07:34:28.075471 30785 sys_catalog.cc:455] T 00000000000000000000000000000000 P 525ffb414e524020ab1bff4ba9ec806f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 525ffb414e524020ab1bff4ba9ec806f. Latest consensus state: current_term: 1 leader_uuid: "525ffb414e524020ab1bff4ba9ec806f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "525ffb414e524020ab1bff4ba9ec806f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40639 } } }
I20260925 07:34:28.075476 30784 sys_catalog.cc:455] T 00000000000000000000000000000000 P 525ffb414e524020ab1bff4ba9ec806f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "525ffb414e524020ab1bff4ba9ec806f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "525ffb414e524020ab1bff4ba9ec806f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40639 } } }
I20260925 07:34:28.075587 30785 sys_catalog.cc:458] T 00000000000000000000000000000000 P 525ffb414e524020ab1bff4ba9ec806f [sys.catalog]: This master's current role is: LEADER
I20260925 07:34:28.075590 30784 sys_catalog.cc:458] T 00000000000000000000000000000000 P 525ffb414e524020ab1bff4ba9ec806f [sys.catalog]: This master's current role is: LEADER
I20260925 07:34:28.076095 30792 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 07:34:28.076609 30792 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 07:34:28.077853 30781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:28.065993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.118:42729 (local address 127.0.0.1:40639)
0925 07:34:28.066184 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:28.066193 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:28.066223 (+ 30us) server_negotiation.cc:407] Connection header received
0925 07:34:28.067118 (+ 895us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:28.067127 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:28.067217 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:28.067422 (+ 205us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:34:28.067950 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:28.069308 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:28.070164 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:28.070723 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:28.070840 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:28.073413 (+ 2573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:34:28.073417 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:34:28.073417 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:34:28.073429 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:34:28.075406 (+ 1977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:34:28.076007 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:34:28.076010 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:34:28.076010 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:34:28.076184 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:34:28.076549 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:34:28.076550 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:34:28.076550 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:34:28.076850 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:34:28.077194 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:28.077574 (+ 380us) server_negotiation.cc:299] Negotiation successful
0925 07:34:28.077673 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":95,"threads_started":1}
I20260925 07:34:28.078758 9480 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 07:34:28.079912 30792 catalog_manager.cc:1347] Generated new cluster ID: 4ba2c962e58846c28ecf3f315dd896aa
I20260925 07:34:28.079978 30792 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 07:34:28.087301 30781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:28.080083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.118:40475 (local address 127.0.0.1:40639)
0925 07:34:28.080118 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:28.080120 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:28.080260 (+ 140us) server_negotiation.cc:407] Connection header received
0925 07:34:28.080309 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:28.080310 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:28.080321 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:28.080392 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:34:28.080945 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:28.081752 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:28.082686 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:28.083016 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:28.083101 (+ 85us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:28.084428 (+ 1327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:34:28.084429 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:34:28.084429 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:34:28.084437 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:34:28.086069 (+ 1632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:34:28.086471 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:34:28.086473 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:34:28.086473 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:34:28.086540 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:34:28.086773 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:34:28.086774 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:34:28.086775 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:34:28.086904 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:34:28.087026 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:28.087143 (+ 117us) server_negotiation.cc:299] Negotiation successful
0925 07:34:28.087210 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 07:34:28.096391 30792 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 07:34:28.096820 30792 catalog_manager.cc:1504] Loading token signing keys...
I20260925 07:34:28.099581 30792 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 525ffb414e524020ab1bff4ba9ec806f: Generated new TSK 0
I20260925 07:34:28.099742 30792 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 07:34:28.104391 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 30710
W20260925 07:34:28.108314 30695 connection.cc:535] client connection to 127.0.0.1:40639 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40639): Connection reset by peer (error 104)
2026-09-25T07:34:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1333 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790321392278483-9480-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 07:34:28 dist-test-slave-57fn krb5kdc[30809](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 07:34:28 dist-test-slave-57fn krb5kdc[30809](info): set up 2 sockets
Sep 25 07:34:28 dist-test-slave-57fn krb5kdc[30809](info): commencing operation
krb5kdc: starting...
W20260925 07:34:29.121783 9480 mini_kdc.cc:121] Time spent starting KDC: real 1.000s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790321392278483-9480-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790321392278483-9480-0/krb5kdc/test-user.keytab.
Sep 25 07:34:29 dist-test-slave-57fn krb5kdc[30809](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321669, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T07:34:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T07:34:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790321392278483-9480-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790321392278483-9480-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 07:34:29.234395 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790321392278483-9480-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790321392278483-9480-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790321392278483-9480-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35727
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790321392278483-9480-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35727
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:34:29.311882 30822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:34:29.312074 30822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:34:29.312096 30822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:34:29.313182 30822 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:34:29.313230 30822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:34:29.313244 30822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:34:29.313256 30822 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:34:29.313268 30822 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:34:29.314615 30822 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790321392278483-9480-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790321392278483-9480-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35727
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790321392278483-9480-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35727
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30822
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790321392278483-9480-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:34:29.314792 30822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:34:29.314949 30822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:34:29.319685 30828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:34:29.319808 30822 server_base.cc:1034] running on GCE node
W20260925 07:34:29.319685 30830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:34:29.319815 30827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:34:29.320147 30822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:34:29.320425 30822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:34:29.321569 30822 hybrid_clock.cc:648] HybridClock initialized: now 1790321669321542 us; error 43 us; skew 500 ppm
Sep 25 07:34:29 dist-test-slave-57fn krb5kdc[30809](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321669, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:34:29.323854 30822 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 07:34:29.324273 30822 webserver.cc:460] Webserver started at http://127.0.0.1:46037/ using document root <none> and password file <none>
I20260925 07:34:29.324522 30822 fs_manager.cc:359] Metadata directory not provided
I20260925 07:34:29.324589 30822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:34:29.324728 30822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:34:29.325834 30822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790321392278483-9480-0/minicluster-data/master-0/data/instance:
uuid: "42e3d700472d4d76af23071b0cce9b90"
format_stamp: "Formatted at 2026-09-25 07:34:29 on dist-test-slave-57fn"
server_key: "fcf0f861fa11aa786d89b005bee7c8ae"
server_key_iv: "832b54d69aacb41c0f288b8af0746c7d"
server_key_version: "encryptionkey@0"
I20260925 07:34:29.326179 30822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790321392278483-9480-0/minicluster-data/master-0/wal/instance:
uuid: "42e3d700472d4d76af23071b0cce9b90"
format_stamp: "Formatted at 2026-09-25 07:34:29 on dist-test-slave-57fn"
server_key: "fcf0f861fa11aa786d89b005bee7c8ae"
server_key_iv: "832b54d69aacb41c0f288b8af0746c7d"
server_key_version: "encryptionkey@0"
I20260925 07:34:29.327556 30822 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 07:34:29.328255 30837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:34:29.328446 30822 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 07:34:29.328545 30822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790321392278483-9480-0/minicluster-data/master-0/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790321392278483-9480-0/minicluster-data/master-0/wal
uuid: "42e3d700472d4d76af23071b0cce9b90"
format_stamp: "Formatted at 2026-09-25 07:34:29 on dist-test-slave-57fn"
server_key: "fcf0f861fa11aa786d89b005bee7c8ae"
server_key_iv: "832b54d69aacb41c0f288b8af0746c7d"
server_key_version: "encryptionkey@0"
I20260925 07:34:29.328622 30822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790321392278483-9480-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790321392278483-9480-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790321392278483-9480-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:34:29.348794 30822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:34:29.350977 30822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:34:29.351140 30822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:34:29.355412 30822 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35727
I20260925 07:34:29.355443 30889 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35727 every 8 connection(s)
I20260925 07:34:29.355777 30822 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
I20260925 07:34:29.356551 30890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:34:29.358536 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 30822
I20260925 07:34:29.358664 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790321392278483-9480-0/minicluster-data/master-0/wal/instance
I20260925 07:34:29.358834 9480 external_mini_cluster.cc:1428] Setting key d6dad24bd03b805247a39a2f94cde284
I20260925 07:34:29.358950 30890 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42e3d700472d4d76af23071b0cce9b90: Bootstrap starting.
I20260925 07:34:29.359697 30890 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 42e3d700472d4d76af23071b0cce9b90: Neither blocks nor log segments found. Creating new log.
I20260925 07:34:29.360054 30890 log.cc:824] T 00000000000000000000000000000000 P 42e3d700472d4d76af23071b0cce9b90: Log is configured to *not* fsync() on all Append() calls
I20260925 07:34:29.360852 30890 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42e3d700472d4d76af23071b0cce9b90: No bootstrap required, opened a new log
I20260925 07:34:29.363133 30890 raft_consensus.cc:348] T 00000000000000000000000000000000 P 42e3d700472d4d76af23071b0cce9b90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42e3d700472d4d76af23071b0cce9b90" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35727 } }
I20260925 07:34:29.363291 30890 raft_consensus.cc:374] T 00000000000000000000000000000000 P 42e3d700472d4d76af23071b0cce9b90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:34:29.363322 30890 raft_consensus.cc:729] T 00000000000000000000000000000000 P 42e3d700472d4d76af23071b0cce9b90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42e3d700472d4d76af23071b0cce9b90, State: Initialized, Role: FOLLOWER
I20260925 07:34:29.363437 30890 consensus_queue.cc:260] T 00000000000000000000000000000000 P 42e3d700472d4d76af23071b0cce9b90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42e3d700472d4d76af23071b0cce9b90" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35727 } }
I20260925 07:34:29.363530 30890 raft_consensus.cc:388] T 00000000000000000000000000000000 P 42e3d700472d4d76af23071b0cce9b90 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 07:34:29.363567 30890 raft_consensus.cc:482] T 00000000000000000000000000000000 P 42e3d700472d4d76af23071b0cce9b90 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 07:34:29.363605 30890 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 42e3d700472d4d76af23071b0cce9b90 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:34:29.364123 30890 raft_consensus.cc:504] T 00000000000000000000000000000000 P 42e3d700472d4d76af23071b0cce9b90 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42e3d700472d4d76af23071b0cce9b90" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35727 } }
I20260925 07:34:29.364252 30890 leader_election.cc:302] T 00000000000000000000000000000000 P 42e3d700472d4d76af23071b0cce9b90 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 42e3d700472d4d76af23071b0cce9b90; no voters:
I20260925 07:34:29.364457 30890 leader_election.cc:288] T 00000000000000000000000000000000 P 42e3d700472d4d76af23071b0cce9b90 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 07:34:29.364481 30895 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 42e3d700472d4d76af23071b0cce9b90 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:34:29.364673 30895 raft_consensus.cc:686] T 00000000000000000000000000000000 P 42e3d700472d4d76af23071b0cce9b90 [term 1 LEADER]: Becoming Leader. State: Replica: 42e3d700472d4d76af23071b0cce9b90, State: Running, Role: LEADER
I20260925 07:34:29.364696 30890 sys_catalog.cc:564] T 00000000000000000000000000000000 P 42e3d700472d4d76af23071b0cce9b90 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:34:29.364846 30895 consensus_queue.cc:237] T 00000000000000000000000000000000 P 42e3d700472d4d76af23071b0cce9b90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42e3d700472d4d76af23071b0cce9b90" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35727 } }
I20260925 07:34:29.365242 30893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:29.359329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.118:55281 (local address 127.0.0.1:35727)
0925 07:34:29.359524 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:29.359533 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:29.359568 (+ 35us) server_negotiation.cc:407] Connection header received
0925 07:34:29.360191 (+ 623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:29.360201 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:29.360262 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:29.360418 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:34:29.360695 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:29.362090 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:29.362883 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:29.363491 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:29.363608 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:29.364423 (+ 815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:34:29.364427 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:34:29.364427 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 07:34:29.364651 (+ 224us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:34:29.364689 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 07:34:29.364703 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:34:29.364984 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:29.365027 (+ 43us) server_negotiation.cc:299] Negotiation successful
0925 07:34:29.365053 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":78,"threads_started":1}
I20260925 07:34:29.365272 30896 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42e3d700472d4d76af23071b0cce9b90 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "42e3d700472d4d76af23071b0cce9b90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42e3d700472d4d76af23071b0cce9b90" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35727 } } }
I20260925 07:34:29.365506 30896 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42e3d700472d4d76af23071b0cce9b90 [sys.catalog]: This master's current role is: LEADER
I20260925 07:34:29.365376 30897 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42e3d700472d4d76af23071b0cce9b90 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 42e3d700472d4d76af23071b0cce9b90. Latest consensus state: current_term: 1 leader_uuid: "42e3d700472d4d76af23071b0cce9b90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42e3d700472d4d76af23071b0cce9b90" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35727 } } }
I20260925 07:34:29.365800 30897 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42e3d700472d4d76af23071b0cce9b90 [sys.catalog]: This master's current role is: LEADER
I20260925 07:34:29.365969 30904 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 07:34:29.366451 30904 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 07:34:29.366461 9480 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 07:34:29.368088 30904 catalog_manager.cc:1347] Generated new cluster ID: f42b7ffe5cff45bdb6045b418c1b2f29
I20260925 07:34:29.368139 30904 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 07:34:29.376351 30893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:29.368056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.118:51515 (local address 127.0.0.1:35727)
0925 07:34:29.368085 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:29.368087 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:29.373699 (+ 5612us) server_negotiation.cc:407] Connection header received
0925 07:34:29.373768 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:29.373768 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:29.373779 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:29.373835 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:34:29.373982 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:29.374459 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:29.375160 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:29.375358 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:29.375428 (+ 70us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:29.376072 (+ 644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:34:29.376073 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:34:29.376073 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 07:34:29.376078 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:34:29.376093 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 07:34:29.376097 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:34:29.376167 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:29.376243 (+ 76us) server_negotiation.cc:299] Negotiation successful
0925 07:34:29.376252 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260925 07:34:29.376365 30904 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 07:34:29.377044 30904 catalog_manager.cc:1504] Loading token signing keys...
I20260925 07:34:29.380447 30904 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 42e3d700472d4d76af23071b0cce9b90: Generated new TSK 0
I20260925 07:34:29.380611 30904 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 07:34:29.385344 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 30822
W20260925 07:34:29.389705 30807 connection.cc:535] client connection to 127.0.0.1:35727 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35727): Connection reset by peer (error 104)
2026-09-25T07:34:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1281 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790321392278483-9480-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 07:34:29 dist-test-slave-57fn krb5kdc[30921](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 07:34:29 dist-test-slave-57fn krb5kdc[30921](info): set up 2 sockets
Sep 25 07:34:29 dist-test-slave-57fn krb5kdc[30921](info): commencing operation
krb5kdc: starting...
W20260925 07:34:30.409651 9480 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790321392278483-9480-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790321392278483-9480-0/krb5kdc/test-user.keytab.
Sep 25 07:34:30 dist-test-slave-57fn krb5kdc[30921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T07:34:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T07:34:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790321392278483-9480-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790321392278483-9480-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 07:34:30.524070 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790321392278483-9480-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790321392278483-9480-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790321392278483-9480-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38733
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790321392278483-9480-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38733
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:34:30.601485 30934 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:34:30.601677 30934 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:34:30.601699 30934 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:34:30.602731 30934 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:34:30.602762 30934 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:34:30.602775 30934 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:34:30.602787 30934 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:34:30.602799 30934 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:34:30.604086 30934 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790321392278483-9480-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790321392278483-9480-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38733
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790321392278483-9480-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38733
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30934
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790321392278483-9480-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:34:30.604264 30934 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:34:30.604408 30934 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:34:30.608733 30940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:34:30.608721 30939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:34:30.608740 30942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:34:30.609119 30934 server_base.cc:1034] running on GCE node
I20260925 07:34:30.609360 30934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:34:30.609668 30934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:34:30.610827 30934 hybrid_clock.cc:648] HybridClock initialized: now 1790321670610821 us; error 46 us; skew 500 ppm
Sep 25 07:34:30 dist-test-slave-57fn krb5kdc[30921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321670, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:34:30.613113 30934 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 07:34:30.613611 30934 webserver.cc:460] Webserver started at http://127.0.0.1:33427/ using document root <none> and password file <none>
I20260925 07:34:30.613827 30934 fs_manager.cc:359] Metadata directory not provided
I20260925 07:34:30.613891 30934 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:34:30.614019 30934 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:34:30.614964 30934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790321392278483-9480-0/minicluster-data/master-0/data/instance:
uuid: "12390bacedad485886f734955f42ee54"
format_stamp: "Formatted at 2026-09-25 07:34:30 on dist-test-slave-57fn"
server_key: "38a3acf69633bc029016f7cae8d32b39"
server_key_iv: "24e9802db70d9487cbe7797643ed39bc"
server_key_version: "encryptionkey@0"
I20260925 07:34:30.615301 30934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790321392278483-9480-0/minicluster-data/master-0/wal/instance:
uuid: "12390bacedad485886f734955f42ee54"
format_stamp: "Formatted at 2026-09-25 07:34:30 on dist-test-slave-57fn"
server_key: "38a3acf69633bc029016f7cae8d32b39"
server_key_iv: "24e9802db70d9487cbe7797643ed39bc"
server_key_version: "encryptionkey@0"
I20260925 07:34:30.616536 30934 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 07:34:30.617183 30949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:34:30.617389 30934 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 07:34:30.617491 30934 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790321392278483-9480-0/minicluster-data/master-0/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790321392278483-9480-0/minicluster-data/master-0/wal
uuid: "12390bacedad485886f734955f42ee54"
format_stamp: "Formatted at 2026-09-25 07:34:30 on dist-test-slave-57fn"
server_key: "38a3acf69633bc029016f7cae8d32b39"
server_key_iv: "24e9802db70d9487cbe7797643ed39bc"
server_key_version: "encryptionkey@0"
I20260925 07:34:30.617592 30934 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790321392278483-9480-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790321392278483-9480-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790321392278483-9480-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:34:30.620554 30934 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:34:30.622308 30934 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:34:30.622457 30934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:34:30.626077 30934 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38733
I20260925 07:34:30.626102 31001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38733 every 8 connection(s)
I20260925 07:34:30.626386 30934 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
I20260925 07:34:30.627386 31002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:34:30.627914 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 30934
I20260925 07:34:30.628017 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790321392278483-9480-0/minicluster-data/master-0/wal/instance
I20260925 07:34:30.628202 9480 external_mini_cluster.cc:1428] Setting key 128986dcbc199628ba3cdde0c2f90113
I20260925 07:34:30.630061 31002 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12390bacedad485886f734955f42ee54: Bootstrap starting.
I20260925 07:34:30.630748 31002 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 12390bacedad485886f734955f42ee54: Neither blocks nor log segments found. Creating new log.
I20260925 07:34:30.631012 31002 log.cc:824] T 00000000000000000000000000000000 P 12390bacedad485886f734955f42ee54: Log is configured to *not* fsync() on all Append() calls
I20260925 07:34:30.631603 31005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:30.628689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.118:52263 (local address 127.0.0.1:38733)
0925 07:34:30.629636 (+ 947us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:30.629645 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:30.629678 (+ 33us) server_negotiation.cc:407] Connection header received
0925 07:34:30.630202 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:30.630209 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:30.630241 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:30.630388 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:34:30.631010 (+ 622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:34:30.631013 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:34:30.631014 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 07:34:30.631018 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:34:30.631054 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 07:34:30.631068 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:34:30.631193 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:30.631405 (+ 212us) server_negotiation.cc:299] Negotiation successful
0925 07:34:30.631453 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":887,"thread_start_us":62,"threads_started":1}
I20260925 07:34:30.631712 31002 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12390bacedad485886f734955f42ee54: No bootstrap required, opened a new log
I20260925 07:34:30.632831 9480 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 07:34:30.634020 31002 raft_consensus.cc:348] T 00000000000000000000000000000000 P 12390bacedad485886f734955f42ee54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12390bacedad485886f734955f42ee54" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38733 } }
I20260925 07:34:30.634162 31002 raft_consensus.cc:374] T 00000000000000000000000000000000 P 12390bacedad485886f734955f42ee54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:34:30.634203 31002 raft_consensus.cc:729] T 00000000000000000000000000000000 P 12390bacedad485886f734955f42ee54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12390bacedad485886f734955f42ee54, State: Initialized, Role: FOLLOWER
I20260925 07:34:30.634364 31002 consensus_queue.cc:260] T 00000000000000000000000000000000 P 12390bacedad485886f734955f42ee54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12390bacedad485886f734955f42ee54" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38733 } }
I20260925 07:34:30.634459 31002 raft_consensus.cc:388] T 00000000000000000000000000000000 P 12390bacedad485886f734955f42ee54 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 07:34:30.634505 31002 raft_consensus.cc:482] T 00000000000000000000000000000000 P 12390bacedad485886f734955f42ee54 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 07:34:30.634562 31002 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 12390bacedad485886f734955f42ee54 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:34:30.635244 31002 raft_consensus.cc:504] T 00000000000000000000000000000000 P 12390bacedad485886f734955f42ee54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12390bacedad485886f734955f42ee54" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38733 } }
I20260925 07:34:30.635340 31005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:30.634244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.118:42161 (local address 127.0.0.1:38733)
0925 07:34:30.634278 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:30.634280 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:30.634392 (+ 112us) server_negotiation.cc:407] Connection header received
0925 07:34:30.634462 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:30.634463 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:30.634468 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:30.634522 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:34:30.635122 (+ 600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:34:30.635122 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:34:30.635123 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 07:34:30.635126 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:34:30.635138 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 07:34:30.635141 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:34:30.635183 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:30.635234 (+ 51us) server_negotiation.cc:299] Negotiation successful
0925 07:34:30.635247 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 07:34:30.635378 31002 leader_election.cc:302] T 00000000000000000000000000000000 P 12390bacedad485886f734955f42ee54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 12390bacedad485886f734955f42ee54; no voters:
I20260925 07:34:30.635548 31002 leader_election.cc:288] T 00000000000000000000000000000000 P 12390bacedad485886f734955f42ee54 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 07:34:30.635603 31012 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 12390bacedad485886f734955f42ee54 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:34:30.635746 31012 raft_consensus.cc:686] T 00000000000000000000000000000000 P 12390bacedad485886f734955f42ee54 [term 1 LEADER]: Becoming Leader. State: Replica: 12390bacedad485886f734955f42ee54, State: Running, Role: LEADER
I20260925 07:34:30.635766 31002 sys_catalog.cc:564] T 00000000000000000000000000000000 P 12390bacedad485886f734955f42ee54 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:34:30.635831 31012 consensus_queue.cc:237] T 00000000000000000000000000000000 P 12390bacedad485886f734955f42ee54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12390bacedad485886f734955f42ee54" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38733 } }
I20260925 07:34:30.636155 31013 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12390bacedad485886f734955f42ee54 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "12390bacedad485886f734955f42ee54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12390bacedad485886f734955f42ee54" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38733 } } }
I20260925 07:34:30.636276 31013 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12390bacedad485886f734955f42ee54 [sys.catalog]: This master's current role is: LEADER
I20260925 07:34:30.636147 31014 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12390bacedad485886f734955f42ee54 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 12390bacedad485886f734955f42ee54. Latest consensus state: current_term: 1 leader_uuid: "12390bacedad485886f734955f42ee54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12390bacedad485886f734955f42ee54" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38733 } } }
I20260925 07:34:30.636461 31014 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12390bacedad485886f734955f42ee54 [sys.catalog]: This master's current role is: LEADER
I20260925 07:34:30.636687 31021 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 07:34:30.637115 31021 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 07:34:30.638535 31021 catalog_manager.cc:1347] Generated new cluster ID: 59f9abc58fdf4de6986f3910766f7a7a
I20260925 07:34:30.638583 31021 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 07:34:30.647826 31021 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 07:34:30.647969 31021 catalog_manager.cc:1504] Loading token signing keys...
I20260925 07:34:30.652527 31021 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 12390bacedad485886f734955f42ee54: Generated new TSK 0
I20260925 07:34:30.652683 31021 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 07:34:30.654615 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 30934
2026-09-25T07:34:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1269 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790321392278483-9480-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 07:34:30 dist-test-slave-57fn krb5kdc[31033](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 07:34:30 dist-test-slave-57fn krb5kdc[31033](info): set up 2 sockets
Sep 25 07:34:30 dist-test-slave-57fn krb5kdc[31033](info): commencing operation
krb5kdc: starting...
W20260925 07:34:31.677559 9480 mini_kdc.cc:121] Time spent starting KDC: real 1.005s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790321392278483-9480-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790321392278483-9480-0/krb5kdc/test-user.keytab.
Sep 25 07:34:31 dist-test-slave-57fn krb5kdc[31033](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321671, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T07:34:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T07:34:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.66.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.66.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.66.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab.
Entry for principal kudu/127.9.66.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab.
I20260925 07:34:31.790509 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790321392278483-9480-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790321392278483-9480-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790321392278483-9480-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.66.62:43791
--webserver_interface=127.9.66.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab
--principal=kudu/127.9.66.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:39459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.66.62:43791
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:34:31.869223 31046 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:34:31.869449 31046 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:34:31.869488 31046 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:34:31.870603 31046 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:34:31.870668 31046 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:34:31.870690 31046 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:34:31.870711 31046 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:34:31.870738 31046 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:34:31.872159 31046 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:39459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790321392278483-9480-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790321392278483-9480-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.66.62:43791
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab
--principal=kudu/127.9.66.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.66.62:43791
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.66.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31046
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790321392278483-9480-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:34:31.872413 31046 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:34:31.872622 31046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:34:31.877230 31051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:34:31.877218 31052 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:34:31.877219 31054 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:34:31.877555 31046 server_base.cc:1034] running on GCE node
I20260925 07:34:31.877786 31046 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:34:31.878072 31046 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:34:31.879258 31046 hybrid_clock.cc:648] HybridClock initialized: now 1790321671879247 us; error 35 us; skew 500 ppm
Sep 25 07:34:31 dist-test-slave-57fn krb5kdc[31033](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321671, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:34:31.881688 31046 init.cc:377] Logged in from keytab as kudu/127.9.66.62@KRBTEST.COM (short username kudu)
I20260925 07:34:31.882103 31046 webserver.cc:460] Webserver started at http://127.9.66.62:43363/ using document root <none> and password file <none>
I20260925 07:34:31.882283 31046 fs_manager.cc:359] Metadata directory not provided
I20260925 07:34:31.882321 31046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:34:31.882411 31046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:34:31.883324 31046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790321392278483-9480-0/minicluster-data/master-0/data/instance:
uuid: "0029146ed8c245b3a460dc9c3828ae31"
format_stamp: "Formatted at 2026-09-25 07:34:31 on dist-test-slave-57fn"
server_key: "2237a26e1c220e190ea01b1affa32ff6"
server_key_iv: "ceb96d1a276229b65907f8a17432c0f9"
server_key_version: "encryptionkey@0"
I20260925 07:34:31.883615 31046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790321392278483-9480-0/minicluster-data/master-0/wal/instance:
uuid: "0029146ed8c245b3a460dc9c3828ae31"
format_stamp: "Formatted at 2026-09-25 07:34:31 on dist-test-slave-57fn"
server_key: "2237a26e1c220e190ea01b1affa32ff6"
server_key_iv: "ceb96d1a276229b65907f8a17432c0f9"
server_key_version: "encryptionkey@0"
I20260925 07:34:31.884807 31046 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 07:34:31.885433 31061 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:34:31.885591 31046 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 07:34:31.885675 31046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790321392278483-9480-0/minicluster-data/master-0/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790321392278483-9480-0/minicluster-data/master-0/wal
uuid: "0029146ed8c245b3a460dc9c3828ae31"
format_stamp: "Formatted at 2026-09-25 07:34:31 on dist-test-slave-57fn"
server_key: "2237a26e1c220e190ea01b1affa32ff6"
server_key_iv: "ceb96d1a276229b65907f8a17432c0f9"
server_key_version: "encryptionkey@0"
I20260925 07:34:31.885723 31046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790321392278483-9480-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790321392278483-9480-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790321392278483-9480-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:34:31.906397 31046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:34:31.908269 31046 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:34:31.908373 31046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:34:31.911907 31046 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.62:43791
I20260925 07:34:31.911964 31113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.62:43791 every 8 connection(s)
I20260925 07:34:31.912253 31046 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
I20260925 07:34:31.913064 31114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:34:31.914786 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 31046
I20260925 07:34:31.914909 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790321392278483-9480-0/minicluster-data/master-0/wal/instance
I20260925 07:34:31.915092 9480 external_mini_cluster.cc:1428] Setting key 081d884436082433248a3130d58905dc
I20260925 07:34:31.915467 31114 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0029146ed8c245b3a460dc9c3828ae31: Bootstrap starting.
I20260925 07:34:31.916047 31114 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0029146ed8c245b3a460dc9c3828ae31: Neither blocks nor log segments found. Creating new log.
I20260925 07:34:31.916314 31114 log.cc:824] T 00000000000000000000000000000000 P 0029146ed8c245b3a460dc9c3828ae31: Log is configured to *not* fsync() on all Append() calls
I20260925 07:34:31.916955 31114 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0029146ed8c245b3a460dc9c3828ae31: No bootstrap required, opened a new log
I20260925 07:34:31.919203 31114 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0029146ed8c245b3a460dc9c3828ae31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0029146ed8c245b3a460dc9c3828ae31" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 43791 } }
I20260925 07:34:31.919353 31114 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0029146ed8c245b3a460dc9c3828ae31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:34:31.919428 31114 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0029146ed8c245b3a460dc9c3828ae31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0029146ed8c245b3a460dc9c3828ae31, State: Initialized, Role: FOLLOWER
I20260925 07:34:31.919585 31114 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0029146ed8c245b3a460dc9c3828ae31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0029146ed8c245b3a460dc9c3828ae31" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 43791 } }
I20260925 07:34:31.919677 31114 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0029146ed8c245b3a460dc9c3828ae31 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 07:34:31.919719 31114 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0029146ed8c245b3a460dc9c3828ae31 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 07:34:31.919766 31114 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0029146ed8c245b3a460dc9c3828ae31 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:34:31.920239 31114 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0029146ed8c245b3a460dc9c3828ae31 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0029146ed8c245b3a460dc9c3828ae31" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 43791 } }
I20260925 07:34:31.920343 31114 leader_election.cc:302] T 00000000000000000000000000000000 P 0029146ed8c245b3a460dc9c3828ae31 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0029146ed8c245b3a460dc9c3828ae31; no voters:
I20260925 07:34:31.920496 31114 leader_election.cc:288] T 00000000000000000000000000000000 P 0029146ed8c245b3a460dc9c3828ae31 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 07:34:31.920532 31119 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0029146ed8c245b3a460dc9c3828ae31 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:34:31.920637 31119 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0029146ed8c245b3a460dc9c3828ae31 [term 1 LEADER]: Becoming Leader. State: Replica: 0029146ed8c245b3a460dc9c3828ae31, State: Running, Role: LEADER
I20260925 07:34:31.920706 31119 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0029146ed8c245b3a460dc9c3828ae31 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0029146ed8c245b3a460dc9c3828ae31" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 43791 } }
I20260925 07:34:31.920727 31114 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0029146ed8c245b3a460dc9c3828ae31 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:34:31.921042 31119 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0029146ed8c245b3a460dc9c3828ae31 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0029146ed8c245b3a460dc9c3828ae31. Latest consensus state: current_term: 1 leader_uuid: "0029146ed8c245b3a460dc9c3828ae31" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0029146ed8c245b3a460dc9c3828ae31" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 43791 } } }
I20260925 07:34:31.921159 31119 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0029146ed8c245b3a460dc9c3828ae31 [sys.catalog]: This master's current role is: LEADER
I20260925 07:34:31.921057 31121 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0029146ed8c245b3a460dc9c3828ae31 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0029146ed8c245b3a460dc9c3828ae31" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0029146ed8c245b3a460dc9c3828ae31" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 43791 } } }
I20260925 07:34:31.921289 31121 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0029146ed8c245b3a460dc9c3828ae31 [sys.catalog]: This master's current role is: LEADER
I20260925 07:34:31.921358 31123 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 07:34:31.921844 31123 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 25 07:34:31 dist-test-slave-57fn krb5kdc[31033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321671, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.66.62@KRBTEST.COM
I20260925 07:34:31.925896 31123 catalog_manager.cc:1347] Generated new cluster ID: 8e6995163d9c4f35bdfbcbac8a2536cc
I20260925 07:34:31.925945 31123 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 07:34:31.926378 31117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:31.915596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52219 (local address 127.9.66.62:43791)
0925 07:34:31.915777 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:31.915785 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:31.915816 (+ 31us) server_negotiation.cc:407] Connection header received
0925 07:34:31.916276 (+ 460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:31.916283 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:31.916326 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:31.916467 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:34:31.917129 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:31.918414 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:31.919187 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:31.919729 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:31.919829 (+ 100us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:31.922729 (+ 2900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:34:31.922733 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:34:31.922733 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:34:31.922744 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:34:31.924692 (+ 1948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:34:31.925076 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:34:31.925078 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:34:31.925079 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:34:31.925173 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:34:31.925364 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:34:31.925365 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:34:31.925365 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:34:31.925673 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:34:31.925895 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:31.925986 (+ 91us) server_negotiation.cc:299] Negotiation successful
0925 07:34:31.926179 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":79,"threads_started":1}
I20260925 07:34:31.927225 9480 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 07:34:31.935189 31123 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 07:34:31.935595 31123 catalog_manager.cc:1504] Loading token signing keys...
I20260925 07:34:31.936309 31117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:31.928645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41705 (local address 127.9.66.62:43791)
0925 07:34:31.928676 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:31.928678 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:31.928976 (+ 298us) server_negotiation.cc:407] Connection header received
0925 07:34:31.929092 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:31.929094 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:31.929116 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:31.929160 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:34:31.929971 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:31.930698 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:31.931749 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:31.932072 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:31.932112 (+ 40us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:31.933562 (+ 1450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:34:31.933563 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:34:31.933563 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:34:31.933570 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:34:31.935097 (+ 1527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:34:31.935480 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:34:31.935481 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:34:31.935482 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:34:31.935533 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:34:31.935736 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:34:31.935737 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:34:31.935737 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:34:31.935838 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:34:31.935942 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:31.936170 (+ 228us) server_negotiation.cc:299] Negotiation successful
0925 07:34:31.936230 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 07:34:31.938802 31123 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0029146ed8c245b3a460dc9c3828ae31: Generated new TSK 0
I20260925 07:34:31.938968 31123 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 07:34:31.940590 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 31046
W20260925 07:34:31.944864 31031 connection.cc:535] client connection to 127.9.66.62:43791 recv error: Network error: failed to read from TLS socket (remote: 127.9.66.62:43791): Connection reset by peer (error 104)
2026-09-25T07:34:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1285 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790321392278483-9480-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 07:34:31 dist-test-slave-57fn krb5kdc[31145](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 07:34:31 dist-test-slave-57fn krb5kdc[31145](info): set up 2 sockets
Sep 25 07:34:31 dist-test-slave-57fn krb5kdc[31145](info): commencing operation
krb5kdc: starting...
W20260925 07:34:32.989698 9480 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790321392278483-9480-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790321392278483-9480-0/krb5kdc/test-user.keytab.
Sep 25 07:34:33 dist-test-slave-57fn krb5kdc[31145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321673, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T07:34:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T07:34:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.66.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.66.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.66.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab.
Entry for principal kudu/127.9.66.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab.
I20260925 07:34:33.100461 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790321392278483-9480-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790321392278483-9480-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790321392278483-9480-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.66.62:37093
--webserver_interface=127.9.66.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab
--principal=kudu/127.9.66.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:38887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.66.62:37093
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:34:33.178356 31158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:34:33.178541 31158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:34:33.178562 31158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:34:33.179711 31158 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:34:33.179754 31158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:34:33.179767 31158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:34:33.179780 31158 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:34:33.179816 31158 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:34:33.181208 31158 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:38887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790321392278483-9480-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790321392278483-9480-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.66.62:37093
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab
--principal=kudu/127.9.66.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.66.62:37093
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.66.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790321392278483-9480-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:34:33.181432 31158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:34:33.181653 31158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 07:34:33.186296 31158 server_base.cc:1034] running on GCE node
W20260925 07:34:33.186240 31166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:34:33.186272 31164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:34:33.186259 31163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:34:33.186733 31158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:34:33.187023 31158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:34:33.188194 31158 hybrid_clock.cc:648] HybridClock initialized: now 1790321673188177 us; error 57 us; skew 500 ppm
Sep 25 07:34:33 dist-test-slave-57fn krb5kdc[31145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321673, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:34:33.195427 31158 init.cc:377] Logged in from keytab as kudu/127.9.66.62@KRBTEST.COM (short username kudu)
I20260925 07:34:33.195868 31158 webserver.cc:460] Webserver started at http://127.9.66.62:41039/ using document root <none> and password file <none>
I20260925 07:34:33.196096 31158 fs_manager.cc:359] Metadata directory not provided
I20260925 07:34:33.196159 31158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:34:33.196296 31158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:34:33.197270 31158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790321392278483-9480-0/minicluster-data/master-0/data/instance:
uuid: "5da7b64dae1346bfa36329f615973804"
format_stamp: "Formatted at 2026-09-25 07:34:33 on dist-test-slave-57fn"
server_key: "3a5023dbad5a0fee6807f958a173a325"
server_key_iv: "2b7b14119d4fe10e4113e2675f170e70"
server_key_version: "encryptionkey@0"
I20260925 07:34:33.197650 31158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790321392278483-9480-0/minicluster-data/master-0/wal/instance:
uuid: "5da7b64dae1346bfa36329f615973804"
format_stamp: "Formatted at 2026-09-25 07:34:33 on dist-test-slave-57fn"
server_key: "3a5023dbad5a0fee6807f958a173a325"
server_key_iv: "2b7b14119d4fe10e4113e2675f170e70"
server_key_version: "encryptionkey@0"
I20260925 07:34:33.198987 31158 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:34:33.199756 31173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:34:33.200003 31158 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:34:33.200102 31158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790321392278483-9480-0/minicluster-data/master-0/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790321392278483-9480-0/minicluster-data/master-0/wal
uuid: "5da7b64dae1346bfa36329f615973804"
format_stamp: "Formatted at 2026-09-25 07:34:33 on dist-test-slave-57fn"
server_key: "3a5023dbad5a0fee6807f958a173a325"
server_key_iv: "2b7b14119d4fe10e4113e2675f170e70"
server_key_version: "encryptionkey@0"
I20260925 07:34:33.200192 31158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790321392278483-9480-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790321392278483-9480-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790321392278483-9480-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:34:33.219918 31158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:34:33.221676 31158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:34:33.221827 31158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:34:33.225852 31158 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.62:37093
I20260925 07:34:33.225893 31225 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.62:37093 every 8 connection(s)
I20260925 07:34:33.226222 31158 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
I20260925 07:34:33.227088 31226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:34:33.229382 31226 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5da7b64dae1346bfa36329f615973804: Bootstrap starting.
I20260925 07:34:33.229986 31226 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5da7b64dae1346bfa36329f615973804: Neither blocks nor log segments found. Creating new log.
I20260925 07:34:33.230237 31226 log.cc:824] T 00000000000000000000000000000000 P 5da7b64dae1346bfa36329f615973804: Log is configured to *not* fsync() on all Append() calls
I20260925 07:34:33.230881 31226 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5da7b64dae1346bfa36329f615973804: No bootstrap required, opened a new log
I20260925 07:34:33.232359 31226 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5da7b64dae1346bfa36329f615973804 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5da7b64dae1346bfa36329f615973804" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 37093 } }
I20260925 07:34:33.232494 31226 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5da7b64dae1346bfa36329f615973804 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:34:33.232519 31226 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5da7b64dae1346bfa36329f615973804 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5da7b64dae1346bfa36329f615973804, State: Initialized, Role: FOLLOWER
I20260925 07:34:33.232616 31226 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5da7b64dae1346bfa36329f615973804 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5da7b64dae1346bfa36329f615973804" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 37093 } }
I20260925 07:34:33.232671 31226 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5da7b64dae1346bfa36329f615973804 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 07:34:33.232693 31226 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5da7b64dae1346bfa36329f615973804 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 07:34:33.232728 31226 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5da7b64dae1346bfa36329f615973804 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:34:33.233453 31226 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5da7b64dae1346bfa36329f615973804 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5da7b64dae1346bfa36329f615973804" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 37093 } }
I20260925 07:34:33.233593 31226 leader_election.cc:302] T 00000000000000000000000000000000 P 5da7b64dae1346bfa36329f615973804 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5da7b64dae1346bfa36329f615973804; no voters:
I20260925 07:34:33.233731 31226 leader_election.cc:288] T 00000000000000000000000000000000 P 5da7b64dae1346bfa36329f615973804 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 07:34:33.233800 31229 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5da7b64dae1346bfa36329f615973804 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:34:33.233968 31229 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5da7b64dae1346bfa36329f615973804 [term 1 LEADER]: Becoming Leader. State: Replica: 5da7b64dae1346bfa36329f615973804, State: Running, Role: LEADER
I20260925 07:34:33.234010 31226 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5da7b64dae1346bfa36329f615973804 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:34:33.234050 31229 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5da7b64dae1346bfa36329f615973804 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5da7b64dae1346bfa36329f615973804" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 37093 } }
I20260925 07:34:33.234375 31230 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5da7b64dae1346bfa36329f615973804 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5da7b64dae1346bfa36329f615973804" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5da7b64dae1346bfa36329f615973804" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 37093 } } }
I20260925 07:34:33.234375 31231 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5da7b64dae1346bfa36329f615973804 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5da7b64dae1346bfa36329f615973804. Latest consensus state: current_term: 1 leader_uuid: "5da7b64dae1346bfa36329f615973804" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5da7b64dae1346bfa36329f615973804" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 37093 } } }
I20260925 07:34:33.234510 31230 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5da7b64dae1346bfa36329f615973804 [sys.catalog]: This master's current role is: LEADER
I20260925 07:34:33.234534 31231 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5da7b64dae1346bfa36329f615973804 [sys.catalog]: This master's current role is: LEADER
I20260925 07:34:33.234699 31236 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 07:34:33.234799 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 31158
I20260925 07:34:33.234925 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790321392278483-9480-0/minicluster-data/master-0/wal/instance
I20260925 07:34:33.235154 9480 external_mini_cluster.cc:1428] Setting key 107a09f1877025c4422dd3728b59890f
I20260925 07:34:33.235235 31236 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 07:34:33.236939 31236 catalog_manager.cc:1347] Generated new cluster ID: 714567ea6566467b89e0182ddb46a246
I20260925 07:34:33.236984 31236 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 07:34:33.240803 31248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:33.235792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54855 (local address 127.9.66.62:37093)
0925 07:34:33.236532 (+ 740us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:33.236539 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:33.236566 (+ 27us) server_negotiation.cc:407] Connection header received
0925 07:34:33.236647 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:33.236652 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:33.236689 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:33.236826 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:34:33.237061 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:33.238050 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:33.238876 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:33.239297 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:33.239433 (+ 136us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:33.240176 (+ 743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:34:33.240180 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:34:33.240180 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 07:34:33.240363 (+ 183us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:34:33.240391 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 07:34:33.240401 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:34:33.240493 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:33.240573 (+ 80us) server_negotiation.cc:299] Negotiation successful
0925 07:34:33.240599 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":703,"thread_start_us":72,"threads_started":1}
I20260925 07:34:33.242162 9480 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 07:34:33.248806 31248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:33.243507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46765 (local address 127.9.66.62:37093)
0925 07:34:33.243545 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:33.243547 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:33.245744 (+ 2197us) server_negotiation.cc:407] Connection header received
0925 07:34:33.245816 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:33.245817 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:33.245828 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:33.245871 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:34:33.246039 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:33.246502 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:33.247464 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:33.247727 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:33.247770 (+ 43us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:33.248543 (+ 773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:34:33.248545 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:34:33.248545 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 07:34:33.248549 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:34:33.248561 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 07:34:33.248565 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:34:33.248596 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:33.248725 (+ 129us) server_negotiation.cc:299] Negotiation successful
0925 07:34:33.248738 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 07:34:33.250824 31236 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 07:34:33.251466 31236 catalog_manager.cc:1504] Loading token signing keys...
I20260925 07:34:33.255257 31236 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5da7b64dae1346bfa36329f615973804: Generated new TSK 0
I20260925 07:34:33.255409 31236 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 07:34:33.257289 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 31158
W20260925 07:34:33.261544 31143 connection.cc:535] client connection to 127.9.66.62:37093 recv error: Network error: failed to read from TLS socket (remote: 127.9.66.62:37093): Connection reset by peer (error 104)
2026-09-25T07:34:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1316 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790321392278483-9480-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 07:34:33 dist-test-slave-57fn krb5kdc[31257](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 07:34:33 dist-test-slave-57fn krb5kdc[31257](info): set up 2 sockets
Sep 25 07:34:33 dist-test-slave-57fn krb5kdc[31257](info): commencing operation
krb5kdc: starting...
W20260925 07:34:34.357250 9480 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790321392278483-9480-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790321392278483-9480-0/krb5kdc/test-user.keytab.
Sep 25 07:34:34 dist-test-slave-57fn krb5kdc[31257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321674, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T07:34:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T07:34:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.66.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.66.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.66.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab.
Entry for principal kudu/127.9.66.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab.
I20260925 07:34:34.472359 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790321392278483-9480-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790321392278483-9480-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790321392278483-9480-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.66.62:35845
--webserver_interface=127.9.66.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab
--principal=kudu/127.9.66.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:43523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.66.62:35845
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:34:34.553861 31270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:34:34.554073 31270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:34:34.554095 31270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:34:34.555176 31270 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:34:34.555210 31270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:34:34.555224 31270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:34:34.555236 31270 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:34:34.555248 31270 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:34:34.556568 31270 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:43523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790321392278483-9480-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790321392278483-9480-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.66.62:35845
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab
--principal=kudu/127.9.66.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.66.62:35845
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.66.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31270
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790321392278483-9480-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:34:34.556741 31270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:34:34.556919 31270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:34:34.561676 31278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:34:34.561745 31270 server_base.cc:1034] running on GCE node
W20260925 07:34:34.561625 31275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:34:34.561623 31276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:34:34.562003 31270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:34:34.562296 31270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:34:34.563450 31270 hybrid_clock.cc:648] HybridClock initialized: now 1790321674563445 us; error 36 us; skew 500 ppm
Sep 25 07:34:34 dist-test-slave-57fn krb5kdc[31257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321674, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:34:34.565804 31270 init.cc:377] Logged in from keytab as kudu/127.9.66.62@KRBTEST.COM (short username kudu)
I20260925 07:34:34.566248 31270 webserver.cc:460] Webserver started at http://127.9.66.62:40741/ using document root <none> and password file <none>
I20260925 07:34:34.566444 31270 fs_manager.cc:359] Metadata directory not provided
I20260925 07:34:34.566496 31270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:34:34.566591 31270 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:34:34.567585 31270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790321392278483-9480-0/minicluster-data/master-0/data/instance:
uuid: "af154812a6a2480fa2bef549c0f2add1"
format_stamp: "Formatted at 2026-09-25 07:34:34 on dist-test-slave-57fn"
server_key: "dbdce826b75ba9e1e38e0274296c17dd"
server_key_iv: "979c30e9dd4f8d18ce41cc27f63b1109"
server_key_version: "encryptionkey@0"
I20260925 07:34:34.567873 31270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790321392278483-9480-0/minicluster-data/master-0/wal/instance:
uuid: "af154812a6a2480fa2bef549c0f2add1"
format_stamp: "Formatted at 2026-09-25 07:34:34 on dist-test-slave-57fn"
server_key: "dbdce826b75ba9e1e38e0274296c17dd"
server_key_iv: "979c30e9dd4f8d18ce41cc27f63b1109"
server_key_version: "encryptionkey@0"
I20260925 07:34:34.569104 31270 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 07:34:34.569821 31285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:34:34.570008 31270 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:34:34.570089 31270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790321392278483-9480-0/minicluster-data/master-0/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790321392278483-9480-0/minicluster-data/master-0/wal
uuid: "af154812a6a2480fa2bef549c0f2add1"
format_stamp: "Formatted at 2026-09-25 07:34:34 on dist-test-slave-57fn"
server_key: "dbdce826b75ba9e1e38e0274296c17dd"
server_key_iv: "979c30e9dd4f8d18ce41cc27f63b1109"
server_key_version: "encryptionkey@0"
I20260925 07:34:34.570143 31270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790321392278483-9480-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790321392278483-9480-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790321392278483-9480-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:34:34.573282 31270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:34:34.575112 31270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:34:34.575218 31270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:34:34.579191 31270 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.62:35845
I20260925 07:34:34.579219 31337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.62:35845 every 8 connection(s)
I20260925 07:34:34.579528 31270 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
I20260925 07:34:34.580402 31338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:34:34.582522 31338 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af154812a6a2480fa2bef549c0f2add1: Bootstrap starting.
I20260925 07:34:34.582988 31338 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P af154812a6a2480fa2bef549c0f2add1: Neither blocks nor log segments found. Creating new log.
I20260925 07:34:34.583202 31338 log.cc:824] T 00000000000000000000000000000000 P af154812a6a2480fa2bef549c0f2add1: Log is configured to *not* fsync() on all Append() calls
I20260925 07:34:34.583778 31338 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af154812a6a2480fa2bef549c0f2add1: No bootstrap required, opened a new log
I20260925 07:34:34.585210 31338 raft_consensus.cc:348] T 00000000000000000000000000000000 P af154812a6a2480fa2bef549c0f2add1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af154812a6a2480fa2bef549c0f2add1" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 35845 } }
I20260925 07:34:34.585336 31338 raft_consensus.cc:374] T 00000000000000000000000000000000 P af154812a6a2480fa2bef549c0f2add1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:34:34.585356 31338 raft_consensus.cc:729] T 00000000000000000000000000000000 P af154812a6a2480fa2bef549c0f2add1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af154812a6a2480fa2bef549c0f2add1, State: Initialized, Role: FOLLOWER
I20260925 07:34:34.585466 31338 consensus_queue.cc:260] T 00000000000000000000000000000000 P af154812a6a2480fa2bef549c0f2add1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af154812a6a2480fa2bef549c0f2add1" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 35845 } }
I20260925 07:34:34.585582 31338 raft_consensus.cc:388] T 00000000000000000000000000000000 P af154812a6a2480fa2bef549c0f2add1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 07:34:34.585608 31338 raft_consensus.cc:482] T 00000000000000000000000000000000 P af154812a6a2480fa2bef549c0f2add1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 07:34:34.585633 31338 raft_consensus.cc:3037] T 00000000000000000000000000000000 P af154812a6a2480fa2bef549c0f2add1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:34:34.586331 31338 raft_consensus.cc:504] T 00000000000000000000000000000000 P af154812a6a2480fa2bef549c0f2add1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af154812a6a2480fa2bef549c0f2add1" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 35845 } }
I20260925 07:34:34.586408 31338 leader_election.cc:302] T 00000000000000000000000000000000 P af154812a6a2480fa2bef549c0f2add1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: af154812a6a2480fa2bef549c0f2add1; no voters:
I20260925 07:34:34.586549 31338 leader_election.cc:288] T 00000000000000000000000000000000 P af154812a6a2480fa2bef549c0f2add1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 07:34:34.586582 31341 raft_consensus.cc:2781] T 00000000000000000000000000000000 P af154812a6a2480fa2bef549c0f2add1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:34:34.586684 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 31270
I20260925 07:34:34.586710 31341 raft_consensus.cc:686] T 00000000000000000000000000000000 P af154812a6a2480fa2bef549c0f2add1 [term 1 LEADER]: Becoming Leader. State: Replica: af154812a6a2480fa2bef549c0f2add1, State: Running, Role: LEADER
I20260925 07:34:34.586794 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790321392278483-9480-0/minicluster-data/master-0/wal/instance
I20260925 07:34:34.586808 31338 sys_catalog.cc:564] T 00000000000000000000000000000000 P af154812a6a2480fa2bef549c0f2add1 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:34:34.586790 31341 consensus_queue.cc:237] T 00000000000000000000000000000000 P af154812a6a2480fa2bef549c0f2add1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af154812a6a2480fa2bef549c0f2add1" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 35845 } }
I20260925 07:34:34.586978 9480 external_mini_cluster.cc:1428] Setting key f1f6c20c9d7183cbc9a4285e03463df7
I20260925 07:34:34.587131 31342 sys_catalog.cc:455] T 00000000000000000000000000000000 P af154812a6a2480fa2bef549c0f2add1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "af154812a6a2480fa2bef549c0f2add1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af154812a6a2480fa2bef549c0f2add1" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 35845 } } }
I20260925 07:34:34.587226 31342 sys_catalog.cc:458] T 00000000000000000000000000000000 P af154812a6a2480fa2bef549c0f2add1 [sys.catalog]: This master's current role is: LEADER
I20260925 07:34:34.587142 31343 sys_catalog.cc:455] T 00000000000000000000000000000000 P af154812a6a2480fa2bef549c0f2add1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader af154812a6a2480fa2bef549c0f2add1. Latest consensus state: current_term: 1 leader_uuid: "af154812a6a2480fa2bef549c0f2add1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af154812a6a2480fa2bef549c0f2add1" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 35845 } } }
I20260925 07:34:34.587276 31343 sys_catalog.cc:458] T 00000000000000000000000000000000 P af154812a6a2480fa2bef549c0f2add1 [sys.catalog]: This master's current role is: LEADER
I20260925 07:34:34.587422 31347 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 07:34:34.587993 31347 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 07:34:34.589545 31353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:34.587717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34529 (local address 127.9.66.62:35845)
0925 07:34:34.587881 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:34.587889 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:34.587921 (+ 32us) server_negotiation.cc:407] Connection header received
0925 07:34:34.588030 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:34.588037 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:34.588065 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:34.588516 (+ 451us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:34:34.588982 (+ 466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:34:34.588985 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:34:34.588985 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 07:34:34.588989 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:34:34.589029 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 07:34:34.589044 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:34:34.589254 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:34.589287 (+ 33us) server_negotiation.cc:299] Negotiation successful
0925 07:34:34.589323 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":73,"threads_started":1}
I20260925 07:34:34.590430 9480 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 07:34:34.590588 31347 catalog_manager.cc:1347] Generated new cluster ID: baeb52229e37485b8dd7ab5dad550f3e
I20260925 07:34:34.590631 31347 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 07:34:34.602604 31353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:34.591756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47309 (local address 127.9.66.62:35845)
0925 07:34:34.591795 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:34.591797 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:34.601646 (+ 9849us) server_negotiation.cc:407] Connection header received
0925 07:34:34.601710 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:34.601710 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:34.601715 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:34.601777 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:34:34.602357 (+ 580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:34:34.602358 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:34:34.602359 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 07:34:34.602363 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:34:34.602376 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 07:34:34.602380 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:34:34.602430 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:34.602500 (+ 70us) server_negotiation.cc:299] Negotiation successful
0925 07:34:34.602513 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 07:34:34.620113 31347 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 07:34:34.620261 31347 catalog_manager.cc:1504] Loading token signing keys...
I20260925 07:34:34.635037 31347 catalog_manager.cc:5868] T 00000000000000000000000000000000 P af154812a6a2480fa2bef549c0f2add1: Generated new TSK 0
I20260925 07:34:34.635208 31347 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 07:34:34.636855 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 31270
2026-09-25T07:34:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1379 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790321392278483-9480-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 07:34:34 dist-test-slave-57fn krb5kdc[31369](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 07:34:34 dist-test-slave-57fn krb5kdc[31369](info): set up 2 sockets
Sep 25 07:34:34 dist-test-slave-57fn krb5kdc[31369](info): commencing operation
krb5kdc: starting...
W20260925 07:34:35.662161 9480 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790321392278483-9480-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790321392278483-9480-0/krb5kdc/test-user.keytab.
Sep 25 07:34:35 dist-test-slave-57fn krb5kdc[31369](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321675, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T07:34:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T07:34:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.66.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.66.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.66.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab.
Entry for principal kudu/127.9.66.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab.
I20260925 07:34:35.781144 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790321392278483-9480-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790321392278483-9480-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790321392278483-9480-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.66.62:43507
--webserver_interface=127.9.66.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab
--principal=kudu/127.9.66.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:44185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.66.62:43507
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:34:35.859532 31382 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:34:35.859759 31382 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:34:35.859797 31382 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:34:35.860924 31382 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:34:35.860988 31382 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:34:35.861014 31382 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:34:35.861043 31382 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:34:35.861065 31382 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:34:35.862521 31382 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:44185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790321392278483-9480-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790321392278483-9480-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.66.62:43507
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab
--principal=kudu/127.9.66.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.66.62:43507
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.66.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31382
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790321392278483-9480-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:34:35.862782 31382 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:34:35.862994 31382 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:34:35.867733 31390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:34:35.867734 31387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:34:35.867765 31388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:34:35.868085 31382 server_base.cc:1034] running on GCE node
I20260925 07:34:35.868348 31382 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:34:35.868644 31382 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:34:35.869845 31382 hybrid_clock.cc:648] HybridClock initialized: now 1790321675869833 us; error 27 us; skew 500 ppm
Sep 25 07:34:35 dist-test-slave-57fn krb5kdc[31369](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321675, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:34:35.872155 31382 init.cc:377] Logged in from keytab as kudu/127.9.66.62@KRBTEST.COM (short username kudu)
I20260925 07:34:35.872571 31382 webserver.cc:460] Webserver started at http://127.9.66.62:43863/ using document root <none> and password file <none>
I20260925 07:34:35.872793 31382 fs_manager.cc:359] Metadata directory not provided
I20260925 07:34:35.872859 31382 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:34:35.873020 31382 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:34:35.874068 31382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790321392278483-9480-0/minicluster-data/master-0/data/instance:
uuid: "a687cf9f57854bb5b2419b7bf452d61a"
format_stamp: "Formatted at 2026-09-25 07:34:35 on dist-test-slave-57fn"
server_key: "06780007f41ef753c1eea44ea9ee0d51"
server_key_iv: "cbbf55e550b1342c2ce33aec50e8686d"
server_key_version: "encryptionkey@0"
I20260925 07:34:35.874413 31382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790321392278483-9480-0/minicluster-data/master-0/wal/instance:
uuid: "a687cf9f57854bb5b2419b7bf452d61a"
format_stamp: "Formatted at 2026-09-25 07:34:35 on dist-test-slave-57fn"
server_key: "06780007f41ef753c1eea44ea9ee0d51"
server_key_iv: "cbbf55e550b1342c2ce33aec50e8686d"
server_key_version: "encryptionkey@0"
I20260925 07:34:35.875686 31382 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 07:34:35.876351 31397 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:34:35.876523 31382 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 07:34:35.876616 31382 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790321392278483-9480-0/minicluster-data/master-0/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790321392278483-9480-0/minicluster-data/master-0/wal
uuid: "a687cf9f57854bb5b2419b7bf452d61a"
format_stamp: "Formatted at 2026-09-25 07:34:35 on dist-test-slave-57fn"
server_key: "06780007f41ef753c1eea44ea9ee0d51"
server_key_iv: "cbbf55e550b1342c2ce33aec50e8686d"
server_key_version: "encryptionkey@0"
I20260925 07:34:35.876689 31382 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790321392278483-9480-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790321392278483-9480-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790321392278483-9480-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:34:35.890056 31382 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:34:35.891943 31382 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:34:35.892099 31382 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:34:35.896018 31382 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.62:43507
I20260925 07:34:35.896049 31449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.62:43507 every 8 connection(s)
I20260925 07:34:35.896369 31382 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
I20260925 07:34:35.897246 31450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:34:35.899538 31450 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a687cf9f57854bb5b2419b7bf452d61a: Bootstrap starting.
I20260925 07:34:35.900106 31450 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a687cf9f57854bb5b2419b7bf452d61a: Neither blocks nor log segments found. Creating new log.
I20260925 07:34:35.900399 31450 log.cc:824] T 00000000000000000000000000000000 P a687cf9f57854bb5b2419b7bf452d61a: Log is configured to *not* fsync() on all Append() calls
I20260925 07:34:35.901146 31450 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a687cf9f57854bb5b2419b7bf452d61a: No bootstrap required, opened a new log
I20260925 07:34:35.902735 31450 raft_consensus.cc:348] T 00000000000000000000000000000000 P a687cf9f57854bb5b2419b7bf452d61a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a687cf9f57854bb5b2419b7bf452d61a" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 43507 } }
I20260925 07:34:35.902870 31450 raft_consensus.cc:374] T 00000000000000000000000000000000 P a687cf9f57854bb5b2419b7bf452d61a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:34:35.902892 31450 raft_consensus.cc:729] T 00000000000000000000000000000000 P a687cf9f57854bb5b2419b7bf452d61a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a687cf9f57854bb5b2419b7bf452d61a, State: Initialized, Role: FOLLOWER
I20260925 07:34:35.903074 31450 consensus_queue.cc:260] T 00000000000000000000000000000000 P a687cf9f57854bb5b2419b7bf452d61a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a687cf9f57854bb5b2419b7bf452d61a" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 43507 } }
I20260925 07:34:35.903156 31450 raft_consensus.cc:388] T 00000000000000000000000000000000 P a687cf9f57854bb5b2419b7bf452d61a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 07:34:35.903194 31450 raft_consensus.cc:482] T 00000000000000000000000000000000 P a687cf9f57854bb5b2419b7bf452d61a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 07:34:35.903241 31450 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a687cf9f57854bb5b2419b7bf452d61a [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:34:35.904042 31450 raft_consensus.cc:504] T 00000000000000000000000000000000 P a687cf9f57854bb5b2419b7bf452d61a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a687cf9f57854bb5b2419b7bf452d61a" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 43507 } }
I20260925 07:34:35.904150 31450 leader_election.cc:302] T 00000000000000000000000000000000 P a687cf9f57854bb5b2419b7bf452d61a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a687cf9f57854bb5b2419b7bf452d61a; no voters:
I20260925 07:34:35.904332 31450 leader_election.cc:288] T 00000000000000000000000000000000 P a687cf9f57854bb5b2419b7bf452d61a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 07:34:35.904433 31453 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a687cf9f57854bb5b2419b7bf452d61a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:34:35.904570 31450 sys_catalog.cc:564] T 00000000000000000000000000000000 P a687cf9f57854bb5b2419b7bf452d61a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:34:35.904590 31453 raft_consensus.cc:686] T 00000000000000000000000000000000 P a687cf9f57854bb5b2419b7bf452d61a [term 1 LEADER]: Becoming Leader. State: Replica: a687cf9f57854bb5b2419b7bf452d61a, State: Running, Role: LEADER
I20260925 07:34:35.904673 31453 consensus_queue.cc:237] T 00000000000000000000000000000000 P a687cf9f57854bb5b2419b7bf452d61a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a687cf9f57854bb5b2419b7bf452d61a" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 43507 } }
I20260925 07:34:35.905015 31455 sys_catalog.cc:455] T 00000000000000000000000000000000 P a687cf9f57854bb5b2419b7bf452d61a [sys.catalog]: SysCatalogTable state changed. Reason: New leader a687cf9f57854bb5b2419b7bf452d61a. Latest consensus state: current_term: 1 leader_uuid: "a687cf9f57854bb5b2419b7bf452d61a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a687cf9f57854bb5b2419b7bf452d61a" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 43507 } } }
I20260925 07:34:35.905112 31455 sys_catalog.cc:458] T 00000000000000000000000000000000 P a687cf9f57854bb5b2419b7bf452d61a [sys.catalog]: This master's current role is: LEADER
I20260925 07:34:35.905289 31454 sys_catalog.cc:455] T 00000000000000000000000000000000 P a687cf9f57854bb5b2419b7bf452d61a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a687cf9f57854bb5b2419b7bf452d61a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a687cf9f57854bb5b2419b7bf452d61a" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 43507 } } }
I20260925 07:34:35.905368 31454 sys_catalog.cc:458] T 00000000000000000000000000000000 P a687cf9f57854bb5b2419b7bf452d61a [sys.catalog]: This master's current role is: LEADER
I20260925 07:34:35.905560 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 31382
I20260925 07:34:35.905670 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790321392278483-9480-0/minicluster-data/master-0/wal/instance
I20260925 07:34:35.905859 9480 external_mini_cluster.cc:1428] Setting key 2c522a2dde34dd79ebc48e6483c4277b
W20260925 07:34:35.906545 31469 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a687cf9f57854bb5b2419b7bf452d61a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 07:34:35.906596 31469 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 07:34:35.906674 31462 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 07:34:35.906829 31462 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 25 07:34:35 dist-test-slave-57fn krb5kdc[31369](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321675, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.66.62@KRBTEST.COM
I20260925 07:34:35.912585 31462 catalog_manager.cc:1347] Generated new cluster ID: ab7a2503d0a24082b7c3065be4f9ad7a
I20260925 07:34:35.912662 31462 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 07:34:35.916147 31472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:35.906289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46213 (local address 127.9.66.62:43507)
0925 07:34:35.906508 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:35.906516 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:35.906551 (+ 35us) server_negotiation.cc:407] Connection header received
0925 07:34:35.906657 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:35.906663 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:35.906713 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:35.906842 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:34:35.907672 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:35.908800 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:35.909677 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:35.910105 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:35.910229 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:35.912556 (+ 2327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:34:35.912560 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:34:35.912560 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:34:35.912571 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:34:35.914560 (+ 1989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:34:35.914959 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:34:35.914962 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:34:35.914962 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:34:35.915053 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:34:35.915250 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:34:35.915251 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:34:35.915251 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:34:35.915518 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:34:35.915655 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:35.915773 (+ 118us) server_negotiation.cc:299] Negotiation successful
0925 07:34:35.915921 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":132,"threads_started":1}
I20260925 07:34:35.916946 9480 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 07:34:35.925333 31472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:35.918434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33147 (local address 127.9.66.62:43507)
0925 07:34:35.918481 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:35.918482 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:35.918703 (+ 221us) server_negotiation.cc:407] Connection header received
0925 07:34:35.918766 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:35.918767 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:35.918777 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:35.918825 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:34:35.919431 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:35.920148 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:35.921128 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:35.921420 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:35.921482 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:35.922756 (+ 1274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:34:35.922757 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:34:35.922757 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:34:35.922763 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:34:35.924262 (+ 1499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:34:35.924627 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:34:35.924628 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:34:35.924628 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:34:35.924676 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:34:35.924871 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:34:35.924872 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:34:35.924872 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:34:35.924989 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:34:35.925067 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:35.925193 (+ 126us) server_negotiation.cc:299] Negotiation successful
0925 07:34:35.925256 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260925 07:34:35.930299 31462 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 07:34:35.930819 31462 catalog_manager.cc:1504] Loading token signing keys...
I20260925 07:34:35.934908 31462 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a687cf9f57854bb5b2419b7bf452d61a: Generated new TSK 0
I20260925 07:34:35.935086 31462 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 07:34:35.940126 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 31382
2026-09-25T07:34:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1303 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790321392278483-9480-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 07:34:35 dist-test-slave-57fn krb5kdc[31481](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 07:34:35 dist-test-slave-57fn krb5kdc[31481](info): set up 2 sockets
Sep 25 07:34:35 dist-test-slave-57fn krb5kdc[31481](info): commencing operation
krb5kdc: starting...
W20260925 07:34:36.957641 9480 mini_kdc.cc:121] Time spent starting KDC: real 1.000s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790321392278483-9480-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790321392278483-9480-0/krb5kdc/test-user.keytab.
Sep 25 07:34:37 dist-test-slave-57fn krb5kdc[31481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321677, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T07:34:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T07:34:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.66.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.66.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.66.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab.
Entry for principal kudu/127.9.66.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab.
I20260925 07:34:37.070165 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790321392278483-9480-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790321392278483-9480-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790321392278483-9480-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.66.62:43055
--webserver_interface=127.9.66.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab
--principal=kudu/127.9.66.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:36945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.66.62:43055
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:34:37.147964 31494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:34:37.148154 31494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:34:37.148177 31494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:34:37.149160 31494 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:34:37.149192 31494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:34:37.149204 31494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:34:37.149215 31494 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:34:37.149230 31494 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:34:37.150630 31494 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:36945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790321392278483-9480-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790321392278483-9480-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.66.62:43055
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab
--principal=kudu/127.9.66.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.66.62:43055
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.66.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31494
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790321392278483-9480-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:34:37.150800 31494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:34:37.150955 31494 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:34:37.155557 31502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:34:37.155683 31494 server_base.cc:1034] running on GCE node
W20260925 07:34:37.155568 31500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:34:37.155643 31499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:34:37.155970 31494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:34:37.156227 31494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:34:37.157400 31494 hybrid_clock.cc:648] HybridClock initialized: now 1790321677157384 us; error 25 us; skew 500 ppm
Sep 25 07:34:37 dist-test-slave-57fn krb5kdc[31481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321677, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:34:37.159898 31494 init.cc:377] Logged in from keytab as kudu/127.9.66.62@KRBTEST.COM (short username kudu)
I20260925 07:34:37.160288 31494 webserver.cc:460] Webserver started at http://127.9.66.62:39941/ using document root <none> and password file <none>
I20260925 07:34:37.160512 31494 fs_manager.cc:359] Metadata directory not provided
I20260925 07:34:37.160580 31494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:34:37.160728 31494 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:34:37.161784 31494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790321392278483-9480-0/minicluster-data/master-0/data/instance:
uuid: "3f2f12bf173544b5a9bcb1ae3b6a1db6"
format_stamp: "Formatted at 2026-09-25 07:34:37 on dist-test-slave-57fn"
server_key: "550551a02dc4af90683f3e07efd3f693"
server_key_iv: "4e4012d5ed6ea5b812899917d29208d4"
server_key_version: "encryptionkey@0"
I20260925 07:34:37.162137 31494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790321392278483-9480-0/minicluster-data/master-0/wal/instance:
uuid: "3f2f12bf173544b5a9bcb1ae3b6a1db6"
format_stamp: "Formatted at 2026-09-25 07:34:37 on dist-test-slave-57fn"
server_key: "550551a02dc4af90683f3e07efd3f693"
server_key_iv: "4e4012d5ed6ea5b812899917d29208d4"
server_key_version: "encryptionkey@0"
I20260925 07:34:37.163410 31494 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 07:34:37.164112 31509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:34:37.164273 31494 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 07:34:37.164366 31494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790321392278483-9480-0/minicluster-data/master-0/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790321392278483-9480-0/minicluster-data/master-0/wal
uuid: "3f2f12bf173544b5a9bcb1ae3b6a1db6"
format_stamp: "Formatted at 2026-09-25 07:34:37 on dist-test-slave-57fn"
server_key: "550551a02dc4af90683f3e07efd3f693"
server_key_iv: "4e4012d5ed6ea5b812899917d29208d4"
server_key_version: "encryptionkey@0"
I20260925 07:34:37.164445 31494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790321392278483-9480-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790321392278483-9480-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790321392278483-9480-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:34:37.181185 31494 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:34:37.182976 31494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:34:37.183128 31494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:34:37.186807 31494 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.62:43055
I20260925 07:34:37.186839 31561 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.62:43055 every 8 connection(s)
I20260925 07:34:37.187175 31494 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
I20260925 07:34:37.188009 31562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:34:37.190158 31562 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f2f12bf173544b5a9bcb1ae3b6a1db6: Bootstrap starting.
I20260925 07:34:37.190651 31562 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3f2f12bf173544b5a9bcb1ae3b6a1db6: Neither blocks nor log segments found. Creating new log.
I20260925 07:34:37.190867 31562 log.cc:824] T 00000000000000000000000000000000 P 3f2f12bf173544b5a9bcb1ae3b6a1db6: Log is configured to *not* fsync() on all Append() calls
I20260925 07:34:37.191470 31562 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f2f12bf173544b5a9bcb1ae3b6a1db6: No bootstrap required, opened a new log
I20260925 07:34:37.192881 31562 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f2f12bf173544b5a9bcb1ae3b6a1db6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f2f12bf173544b5a9bcb1ae3b6a1db6" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 43055 } }
I20260925 07:34:37.193014 31562 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3f2f12bf173544b5a9bcb1ae3b6a1db6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:34:37.193037 31562 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f2f12bf173544b5a9bcb1ae3b6a1db6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f2f12bf173544b5a9bcb1ae3b6a1db6, State: Initialized, Role: FOLLOWER
I20260925 07:34:37.193169 31562 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f2f12bf173544b5a9bcb1ae3b6a1db6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f2f12bf173544b5a9bcb1ae3b6a1db6" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 43055 } }
I20260925 07:34:37.193260 31562 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3f2f12bf173544b5a9bcb1ae3b6a1db6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 07:34:37.193282 31562 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3f2f12bf173544b5a9bcb1ae3b6a1db6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 07:34:37.193310 31562 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f2f12bf173544b5a9bcb1ae3b6a1db6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:34:37.194177 31562 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3f2f12bf173544b5a9bcb1ae3b6a1db6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f2f12bf173544b5a9bcb1ae3b6a1db6" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 43055 } }
I20260925 07:34:37.194259 31562 leader_election.cc:302] T 00000000000000000000000000000000 P 3f2f12bf173544b5a9bcb1ae3b6a1db6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3f2f12bf173544b5a9bcb1ae3b6a1db6; no voters:
I20260925 07:34:37.194413 31562 leader_election.cc:288] T 00000000000000000000000000000000 P 3f2f12bf173544b5a9bcb1ae3b6a1db6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 07:34:37.194458 31565 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3f2f12bf173544b5a9bcb1ae3b6a1db6 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:34:37.194617 31565 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3f2f12bf173544b5a9bcb1ae3b6a1db6 [term 1 LEADER]: Becoming Leader. State: Replica: 3f2f12bf173544b5a9bcb1ae3b6a1db6, State: Running, Role: LEADER
I20260925 07:34:37.194617 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 31494
I20260925 07:34:37.194669 31562 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f2f12bf173544b5a9bcb1ae3b6a1db6 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:34:37.194736 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790321392278483-9480-0/minicluster-data/master-0/wal/instance
I20260925 07:34:37.194795 31565 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3f2f12bf173544b5a9bcb1ae3b6a1db6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f2f12bf173544b5a9bcb1ae3b6a1db6" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 43055 } }
I20260925 07:34:37.194960 9480 external_mini_cluster.cc:1428] Setting key 7f2f7b8a07ee85ba4215142dc5f9dcb9
I20260925 07:34:37.195164 31565 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f2f12bf173544b5a9bcb1ae3b6a1db6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f2f12bf173544b5a9bcb1ae3b6a1db6. Latest consensus state: current_term: 1 leader_uuid: "3f2f12bf173544b5a9bcb1ae3b6a1db6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f2f12bf173544b5a9bcb1ae3b6a1db6" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 43055 } } }
I20260925 07:34:37.195578 31565 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f2f12bf173544b5a9bcb1ae3b6a1db6 [sys.catalog]: This master's current role is: LEADER
I20260925 07:34:37.195683 31566 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f2f12bf173544b5a9bcb1ae3b6a1db6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3f2f12bf173544b5a9bcb1ae3b6a1db6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f2f12bf173544b5a9bcb1ae3b6a1db6" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 43055 } } }
I20260925 07:34:37.195822 31566 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f2f12bf173544b5a9bcb1ae3b6a1db6 [sys.catalog]: This master's current role is: LEADER
I20260925 07:34:37.196013 31577 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 07:34:37.196794 31577 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 07:34:37.198253 31577 catalog_manager.cc:1347] Generated new cluster ID: de80f1109d9b4570b317ada1b45670f9
I20260925 07:34:37.198305 31577 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 07:34:37.201586 31575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:37.195455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60017 (local address 127.9.66.62:43055)
0925 07:34:37.195874 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:37.195885 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:37.196293 (+ 408us) server_negotiation.cc:407] Connection header received
0925 07:34:37.196402 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:37.196407 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:37.196451 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:37.196561 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:34:37.196840 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:37.198188 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:37.199344 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:37.199977 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:37.200085 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:37.200798 (+ 713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:34:37.200802 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:34:37.200802 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 07:34:37.201006 (+ 204us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:34:37.201042 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 07:34:37.201056 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:34:37.201124 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:37.201204 (+ 80us) server_negotiation.cc:299] Negotiation successful
0925 07:34:37.201354 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":72,"threads_started":1}
I20260925 07:34:37.202498 9480 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 07:34:37.207269 31575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:37.203906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54541 (local address 127.9.66.62:43055)
0925 07:34:37.203939 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:37.203941 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:37.204046 (+ 105us) server_negotiation.cc:407] Connection header received
0925 07:34:37.204091 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:37.204091 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:37.204100 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:37.204158 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:34:37.204276 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:37.204993 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:37.205994 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:37.206284 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:37.206381 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:37.207006 (+ 625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:34:37.207007 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:34:37.207007 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 07:34:37.207012 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:34:37.207026 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 07:34:37.207030 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:34:37.207092 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:37.207176 (+ 84us) server_negotiation.cc:299] Negotiation successful
0925 07:34:37.207202 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 07:34:37.210458 31577 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 07:34:37.211124 31577 catalog_manager.cc:1504] Loading token signing keys...
I20260925 07:34:37.217223 31577 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3f2f12bf173544b5a9bcb1ae3b6a1db6: Generated new TSK 0
I20260925 07:34:37.217435 31577 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 07:34:37.219432 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 31494
2026-09-25T07:34:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1279 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790321392278483-9480-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 07:34:37 dist-test-slave-57fn krb5kdc[31593](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 07:34:37 dist-test-slave-57fn krb5kdc[31593](info): set up 2 sockets
Sep 25 07:34:37 dist-test-slave-57fn krb5kdc[31593](info): commencing operation
krb5kdc: starting...
W20260925 07:34:38.249747 9480 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790321392278483-9480-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790321392278483-9480-0/krb5kdc/test-user.keytab.
Sep 25 07:34:38 dist-test-slave-57fn krb5kdc[31593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321678, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T07:34:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T07:34:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.66.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.66.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.66.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab.
Entry for principal kudu/127.9.66.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab.
I20260925 07:34:38.362058 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790321392278483-9480-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790321392278483-9480-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790321392278483-9480-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.66.62:40061
--webserver_interface=127.9.66.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab
--principal=kudu/127.9.66.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:37639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.66.62:40061
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:34:38.440500 31606 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:34:38.440725 31606 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:34:38.440760 31606 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:34:38.441910 31606 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:34:38.441965 31606 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:34:38.441987 31606 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:34:38.442008 31606 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:34:38.442035 31606 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:34:38.443367 31606 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:37639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790321392278483-9480-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790321392278483-9480-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.66.62:40061
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab
--principal=kudu/127.9.66.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.66.62:40061
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.66.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31606
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790321392278483-9480-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:34:38.443624 31606 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:34:38.443809 31606 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:34:38.448307 31612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:34:38.448680 31614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:34:38.448809 31611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:34:38.448967 31606 server_base.cc:1034] running on GCE node
I20260925 07:34:38.449184 31606 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:34:38.449471 31606 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:34:38.450663 31606 hybrid_clock.cc:648] HybridClock initialized: now 1790321678450653 us; error 28 us; skew 500 ppm
Sep 25 07:34:38 dist-test-slave-57fn krb5kdc[31593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321678, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:34:38.459270 31606 init.cc:377] Logged in from keytab as kudu/127.9.66.62@KRBTEST.COM (short username kudu)
I20260925 07:34:38.459687 31606 webserver.cc:460] Webserver started at http://127.9.66.62:41875/ using document root <none> and password file <none>
I20260925 07:34:38.459923 31606 fs_manager.cc:359] Metadata directory not provided
I20260925 07:34:38.459995 31606 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:34:38.460137 31606 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:34:38.461143 31606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790321392278483-9480-0/minicluster-data/master-0/data/instance:
uuid: "a36d91f56e7d479fa79ffa6436d07404"
format_stamp: "Formatted at 2026-09-25 07:34:38 on dist-test-slave-57fn"
server_key: "62a5a5f337e0d70e9027079c9f554833"
server_key_iv: "18273badf06c75b50138985cc9668e5c"
server_key_version: "encryptionkey@0"
I20260925 07:34:38.461484 31606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790321392278483-9480-0/minicluster-data/master-0/wal/instance:
uuid: "a36d91f56e7d479fa79ffa6436d07404"
format_stamp: "Formatted at 2026-09-25 07:34:38 on dist-test-slave-57fn"
server_key: "62a5a5f337e0d70e9027079c9f554833"
server_key_iv: "18273badf06c75b50138985cc9668e5c"
server_key_version: "encryptionkey@0"
I20260925 07:34:38.462755 31606 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:34:38.463560 31621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:34:38.463723 31606 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 07:34:38.463812 31606 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790321392278483-9480-0/minicluster-data/master-0/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790321392278483-9480-0/minicluster-data/master-0/wal
uuid: "a36d91f56e7d479fa79ffa6436d07404"
format_stamp: "Formatted at 2026-09-25 07:34:38 on dist-test-slave-57fn"
server_key: "62a5a5f337e0d70e9027079c9f554833"
server_key_iv: "18273badf06c75b50138985cc9668e5c"
server_key_version: "encryptionkey@0"
I20260925 07:34:38.463892 31606 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790321392278483-9480-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790321392278483-9480-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790321392278483-9480-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:34:38.467070 31606 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:34:38.469353 31606 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:34:38.469566 31606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:34:38.473213 31606 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.62:40061
I20260925 07:34:38.473243 31673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.62:40061 every 8 connection(s)
I20260925 07:34:38.473610 31606 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
I20260925 07:34:38.474534 31674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:34:38.475917 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 31606
I20260925 07:34:38.476050 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790321392278483-9480-0/minicluster-data/master-0/wal/instance
I20260925 07:34:38.476217 9480 external_mini_cluster.cc:1428] Setting key 488f8fd91dcafd24ba0d2db6b57f6219
I20260925 07:34:38.476940 31674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a36d91f56e7d479fa79ffa6436d07404: Bootstrap starting.
I20260925 07:34:38.477581 31674 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a36d91f56e7d479fa79ffa6436d07404: Neither blocks nor log segments found. Creating new log.
I20260925 07:34:38.477933 31674 log.cc:824] T 00000000000000000000000000000000 P a36d91f56e7d479fa79ffa6436d07404: Log is configured to *not* fsync() on all Append() calls
I20260925 07:34:38.478420 31677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:38.476708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49657 (local address 127.9.66.62:40061)
0925 07:34:38.476868 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:38.476876 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:38.476907 (+ 31us) server_negotiation.cc:407] Connection header received
0925 07:34:38.477328 (+ 421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:38.477334 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:38.477356 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:38.477467 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:34:38.478040 (+ 573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:34:38.478043 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:34:38.478043 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 07:34:38.478047 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:34:38.478082 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 07:34:38.478092 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:34:38.478178 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:38.478244 (+ 66us) server_negotiation.cc:299] Negotiation successful
0925 07:34:38.478282 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":83,"threads_started":1}
I20260925 07:34:38.478694 31674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a36d91f56e7d479fa79ffa6436d07404: No bootstrap required, opened a new log
I20260925 07:34:38.479373 9480 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 07:34:38.480708 31674 raft_consensus.cc:348] T 00000000000000000000000000000000 P a36d91f56e7d479fa79ffa6436d07404 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a36d91f56e7d479fa79ffa6436d07404" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 40061 } }
I20260925 07:34:38.480870 31674 raft_consensus.cc:374] T 00000000000000000000000000000000 P a36d91f56e7d479fa79ffa6436d07404 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 07:34:38.480928 31674 raft_consensus.cc:729] T 00000000000000000000000000000000 P a36d91f56e7d479fa79ffa6436d07404 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a36d91f56e7d479fa79ffa6436d07404, State: Initialized, Role: FOLLOWER
I20260925 07:34:38.481062 31674 consensus_queue.cc:260] T 00000000000000000000000000000000 P a36d91f56e7d479fa79ffa6436d07404 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a36d91f56e7d479fa79ffa6436d07404" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 40061 } }
I20260925 07:34:38.481166 31674 raft_consensus.cc:388] T 00000000000000000000000000000000 P a36d91f56e7d479fa79ffa6436d07404 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 07:34:38.481223 31674 raft_consensus.cc:482] T 00000000000000000000000000000000 P a36d91f56e7d479fa79ffa6436d07404 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 07:34:38.481272 31674 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a36d91f56e7d479fa79ffa6436d07404 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:34:38.481798 31677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:38.480803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33757 (local address 127.9.66.62:40061)
0925 07:34:38.480836 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:38.480838 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:38.480938 (+ 100us) server_negotiation.cc:407] Connection header received
0925 07:34:38.480967 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:38.480967 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:38.480972 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:38.481029 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:34:38.481600 (+ 571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:34:38.481601 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:34:38.481602 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 07:34:38.481605 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:34:38.481617 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 07:34:38.481620 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:34:38.481674 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:38.481729 (+ 55us) server_negotiation.cc:299] Negotiation successful
0925 07:34:38.481742 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260925 07:34:38.481873 31674 raft_consensus.cc:504] T 00000000000000000000000000000000 P a36d91f56e7d479fa79ffa6436d07404 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a36d91f56e7d479fa79ffa6436d07404" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 40061 } }
I20260925 07:34:38.481982 31674 leader_election.cc:302] T 00000000000000000000000000000000 P a36d91f56e7d479fa79ffa6436d07404 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a36d91f56e7d479fa79ffa6436d07404; no voters:
I20260925 07:34:38.482131 31674 leader_election.cc:288] T 00000000000000000000000000000000 P a36d91f56e7d479fa79ffa6436d07404 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 07:34:38.482226 31684 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a36d91f56e7d479fa79ffa6436d07404 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:34:38.482329 31684 raft_consensus.cc:686] T 00000000000000000000000000000000 P a36d91f56e7d479fa79ffa6436d07404 [term 1 LEADER]: Becoming Leader. State: Replica: a36d91f56e7d479fa79ffa6436d07404, State: Running, Role: LEADER
I20260925 07:34:38.482368 31674 sys_catalog.cc:564] T 00000000000000000000000000000000 P a36d91f56e7d479fa79ffa6436d07404 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:34:38.482476 31684 consensus_queue.cc:237] T 00000000000000000000000000000000 P a36d91f56e7d479fa79ffa6436d07404 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a36d91f56e7d479fa79ffa6436d07404" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 40061 } }
I20260925 07:34:38.482894 31686 sys_catalog.cc:455] T 00000000000000000000000000000000 P a36d91f56e7d479fa79ffa6436d07404 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a36d91f56e7d479fa79ffa6436d07404. Latest consensus state: current_term: 1 leader_uuid: "a36d91f56e7d479fa79ffa6436d07404" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a36d91f56e7d479fa79ffa6436d07404" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 40061 } } }
I20260925 07:34:38.482877 31685 sys_catalog.cc:455] T 00000000000000000000000000000000 P a36d91f56e7d479fa79ffa6436d07404 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a36d91f56e7d479fa79ffa6436d07404" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a36d91f56e7d479fa79ffa6436d07404" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 40061 } } }
I20260925 07:34:38.483074 31685 sys_catalog.cc:458] T 00000000000000000000000000000000 P a36d91f56e7d479fa79ffa6436d07404 [sys.catalog]: This master's current role is: LEADER
I20260925 07:34:38.483456 31686 sys_catalog.cc:458] T 00000000000000000000000000000000 P a36d91f56e7d479fa79ffa6436d07404 [sys.catalog]: This master's current role is: LEADER
W20260925 07:34:38.484064 31700 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a36d91f56e7d479fa79ffa6436d07404: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 07:34:38.484124 31700 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 07:34:38.484169 31693 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 07:34:38.484313 31693 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 07:34:38.485653 31693 catalog_manager.cc:1347] Generated new cluster ID: baf3d16341aa4d0993af781cc6e249ac
I20260925 07:34:38.485706 31693 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 07:34:38.505237 31693 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 07:34:38.505403 31693 catalog_manager.cc:1504] Loading token signing keys...
I20260925 07:34:38.509204 31693 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a36d91f56e7d479fa79ffa6436d07404: Generated new TSK 0
I20260925 07:34:38.509394 31693 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 07:34:38.513367 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 31606
2026-09-25T07:34:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1293 ms)
[----------] 18 tests from AuthTokenIssuingTest (23434 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790321392278483-9480-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 07:34:38 dist-test-slave-57fn krb5kdc[31705](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 07:34:38 dist-test-slave-57fn krb5kdc[31705](info): set up 2 sockets
Sep 25 07:34:38 dist-test-slave-57fn krb5kdc[31705](info): commencing operation
krb5kdc: starting...
W20260925 07:34:39.517831 9480 mini_kdc.cc:121] Time spent starting KDC: real 0.987s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790321392278483-9480-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790321392278483-9480-0/krb5kdc/test-user.keytab.
Sep 25 07:34:39 dist-test-slave-57fn krb5kdc[31705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321679, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T07:34:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T07:34:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.66.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.66.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.66.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab.
Entry for principal kudu/127.9.66.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab.
I20260925 07:34:39.632493 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790321392278483-9480-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790321392278483-9480-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790321392278483-9480-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.66.62:40975
--webserver_interface=127.9.66.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab
--principal=kudu/127.9.66.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:42063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.66.62:40975,127.9.66.61:42749,127.9.66.60:36157
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:34:39.709496 31718 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 07:34:39.709766 31718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:34:39.709794 31718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:34:39.709839 31718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:34:39.710955 31718 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:34:39.711016 31718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:34:39.711040 31718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:34:39.711061 31718 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:34:39.711091 31718 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:34:39.712489 31718 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:42063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790321392278483-9480-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790321392278483-9480-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.66.62:40975,127.9.66.61:42749,127.9.66.60:36157
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab
--principal=kudu/127.9.66.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.66.62:40975
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.66.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31718
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790321392278483-9480-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:34:39.712749 31718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:34:39.712954 31718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:34:39.717419 31726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:34:39.717579 31718 server_base.cc:1034] running on GCE node
W20260925 07:34:39.717391 31724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:34:39.717626 31723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:34:39.717880 31718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:34:39.718215 31718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:34:39.719393 31718 hybrid_clock.cc:648] HybridClock initialized: now 1790321679719369 us; error 52 us; skew 500 ppm
Sep 25 07:34:39 dist-test-slave-57fn krb5kdc[31705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321679, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:34:39.721740 31718 init.cc:377] Logged in from keytab as kudu/127.9.66.62@KRBTEST.COM (short username kudu)
I20260925 07:34:39.722167 31718 webserver.cc:460] Webserver started at http://127.9.66.62:41591/ using document root <none> and password file <none>
I20260925 07:34:39.722344 31718 fs_manager.cc:359] Metadata directory not provided
I20260925 07:34:39.722380 31718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:34:39.722460 31718 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:34:39.723376 31718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790321392278483-9480-0/minicluster-data/master-0/data/instance:
uuid: "88eae5eabfc64f108cc52d88079bec35"
format_stamp: "Formatted at 2026-09-25 07:34:39 on dist-test-slave-57fn"
server_key: "9b75c9b3422b3adc767c7fc102dd7b2f"
server_key_iv: "60d2a818fa67d289fbcf1348e5d3cf81"
server_key_version: "encryptionkey@0"
I20260925 07:34:39.723651 31718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790321392278483-9480-0/minicluster-data/master-0/wal/instance:
uuid: "88eae5eabfc64f108cc52d88079bec35"
format_stamp: "Formatted at 2026-09-25 07:34:39 on dist-test-slave-57fn"
server_key: "9b75c9b3422b3adc767c7fc102dd7b2f"
server_key_iv: "60d2a818fa67d289fbcf1348e5d3cf81"
server_key_version: "encryptionkey@0"
I20260925 07:34:39.724781 31718 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 07:34:39.725448 31733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:34:39.725672 31718 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 07:34:39.725755 31718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790321392278483-9480-0/minicluster-data/master-0/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790321392278483-9480-0/minicluster-data/master-0/wal
uuid: "88eae5eabfc64f108cc52d88079bec35"
format_stamp: "Formatted at 2026-09-25 07:34:39 on dist-test-slave-57fn"
server_key: "9b75c9b3422b3adc767c7fc102dd7b2f"
server_key_iv: "60d2a818fa67d289fbcf1348e5d3cf81"
server_key_version: "encryptionkey@0"
I20260925 07:34:39.725805 31718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790321392278483-9480-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790321392278483-9480-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790321392278483-9480-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:34:39.738591 31718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:34:39.740679 31718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:34:39.740780 31718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:34:39.744282 31718 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.62:40975
I20260925 07:34:39.744329 31785 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.62:40975 every 8 connection(s)
I20260925 07:34:39.744648 31718 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
I20260925 07:34:39.745620 31786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:34:39.746685 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 31718
I20260925 07:34:39.746773 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790321392278483-9480-0/minicluster-data/master-0/wal/instance
I20260925 07:34:39.746989 9480 external_mini_cluster.cc:1428] Setting key b15fe399680110f65c5655eb28f75105
I20260925 07:34:39.747166 31786 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.66.62" port: 40975 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 07:34:39 dist-test-slave-57fn krb5kdc[31705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321679, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.62@KRBTEST.COM for kudu/127.9.66.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.66.61@KRBTEST.COM; defaulting to no policy
I20260925 07:34:39.756340 31788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:39.747779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:40975 (local address 127.0.0.1:42828)
0925 07:34:39.747942 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 07:34:39.747981 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 07:34:39.748438 (+ 457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:34:39.748716 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:34:39.748719 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:34:39.748971 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:34:39.749109 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:39.749114 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:39.750207 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:39.750209 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:39.750864 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:39.750866 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:39.751376 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:39.751377 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:39.751439 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:39.751952 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:34:39.751959 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:34:39.753230 (+ 1271us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:34:39.754929 (+ 1699us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:34:39.754935 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:34:39.754937 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:34:39.755124 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:34:39.755357 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:34:39.755357 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:34:39.755358 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:34:39.755433 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:34:39.755790 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:34:39.755795 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:34:39.755853 (+ 58us) client_negotiation.cc:769] Sending connection context
0925 07:34:39.755955 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 07:34:39.756180 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"mutex_wait_us":97,"thread_start_us":95,"threads_started":1}
I20260925 07:34:39.756343 31789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:39.747803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42828 (local address 127.9.66.62:40975)
0925 07:34:39.748009 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:39.748014 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:39.748077 (+ 63us) server_negotiation.cc:407] Connection header received
0925 07:34:39.748587 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:39.748590 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:39.748609 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:39.748713 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:34:39.749205 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:39.750114 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:39.750957 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:39.751259 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:39.751387 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:39.753316 (+ 1929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:34:39.753322 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:34:39.753322 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:34:39.753329 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:34:39.754835 (+ 1506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:34:39.755212 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:34:39.755215 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:34:39.755215 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:34:39.755290 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:34:39.755495 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:34:39.755495 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:34:39.755496 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:34:39.755704 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:34:39.755811 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:39.755979 (+ 168us) server_negotiation.cc:299] Negotiation successful
0925 07:34:39.756118 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":90,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.66.61@KRBTEST.COM" created.
I20260925 07:34:39.757093 31786 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.66.61" port: 42749 } has no permanent_uuid. Determining permanent_uuid...
I20260925 07:34:39.757448 31788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:39.757326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.61:42749 (local address 127.0.0.1:47560)
0925 07:34:39.757372 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0925 07:34:39.757417 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.61:42749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260925 07:34:39.757606 31735 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.66.61:42749: connect: Connection refused (error 111)
I20260925 07:34:39.758068 31788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:39.757943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.61:42749 (local address 127.0.0.1:47574)
0925 07:34:39.758009 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0925 07:34:39.758040 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.61:42749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260925 07:34:39.758203 31786 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.66.61:42749: Network error: Client connection negotiation failed: client connection to 127.9.66.61:42749: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.66.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.61.keytab.
Entry for principal kudu/127.9.66.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.61.keytab.
I20260925 07:34:39.772558 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790321392278483-9480-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790321392278483-9480-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790321392278483-9480-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790321392278483-9480-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.66.61:42749
--webserver_interface=127.9.66.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.61.keytab
--principal=kudu/127.9.66.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:42063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.66.62:40975,127.9.66.61:42749,127.9.66.60:36157
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 07:34:39.807402 31786 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.66.61" port: 42749 } attempt: 1
I20260925 07:34:39.807857 31788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:39.807729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.61:42749 (local address 127.0.0.1:47586)
0925 07:34:39.807791 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0925 07:34:39.807828 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.61:42749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260925 07:34:39.808301 31788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:39.808212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.61:42749 (local address 127.0.0.1:47594)
0925 07:34:39.808259 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0925 07:34:39.808284 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.61:42749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260925 07:34:39.808439 31786 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.66.61:42749: Network error: Client connection negotiation failed: client connection to 127.9.66.61:42749: connect: Connection refused (error 111)
W20260925 07:34:39.852643 31793 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 07:34:39.852859 31793 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:34:39.852888 31793 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:34:39.852927 31793 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:34:39.854056 31793 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:34:39.854112 31793 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:34:39.854136 31793 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:34:39.854164 31793 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:34:39.854187 31793 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:34:39.855619 31793 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:42063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790321392278483-9480-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790321392278483-9480-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.9.66.62:40975,127.9.66.61:42749,127.9.66.60:36157
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.61.keytab
--principal=kudu/127.9.66.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.66.61:42749
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790321392278483-9480-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.9.66.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31793
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790321392278483-9480-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:34:39.855868 31793 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:34:39.856076 31793 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:34:39.861151 31801 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:34:39.861182 31798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:34:39.861115 31799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:34:39.861279 31793 server_base.cc:1034] running on GCE node
I20260925 07:34:39.861482 31793 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:34:39.861836 31793 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:34:39.863010 31793 hybrid_clock.cc:648] HybridClock initialized: now 1790321679863010 us; error 41 us; skew 500 ppm
Sep 25 07:34:39 dist-test-slave-57fn krb5kdc[31705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321679, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:34:39.865590 31793 init.cc:377] Logged in from keytab as kudu/127.9.66.61@KRBTEST.COM (short username kudu)
I20260925 07:34:39.865991 31793 webserver.cc:460] Webserver started at http://127.9.66.61:41787/ using document root <none> and password file <none>
I20260925 07:34:39.866211 31793 fs_manager.cc:359] Metadata directory not provided
I20260925 07:34:39.866276 31793 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:34:39.866426 31793 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:34:39.867440 31793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790321392278483-9480-0/minicluster-data/master-1/data/instance:
uuid: "8293e1e0553246bcb73add0eaed8eddd"
format_stamp: "Formatted at 2026-09-25 07:34:39 on dist-test-slave-57fn"
server_key: "0f7d92be45828144d03a83d3bdd776c2"
server_key_iv: "ad3f2f85017e6d44361c5aa2b55e752b"
server_key_version: "encryptionkey@0"
I20260925 07:34:39.867792 31793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790321392278483-9480-0/minicluster-data/master-1/wal/instance:
uuid: "8293e1e0553246bcb73add0eaed8eddd"
format_stamp: "Formatted at 2026-09-25 07:34:39 on dist-test-slave-57fn"
server_key: "0f7d92be45828144d03a83d3bdd776c2"
server_key_iv: "ad3f2f85017e6d44361c5aa2b55e752b"
server_key_version: "encryptionkey@0"
I20260925 07:34:39.869046 31793 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 07:34:39.869779 31808 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:34:39.869940 31793 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 07:34:39.870031 31793 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790321392278483-9480-0/minicluster-data/master-1/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790321392278483-9480-0/minicluster-data/master-1/wal
uuid: "8293e1e0553246bcb73add0eaed8eddd"
format_stamp: "Formatted at 2026-09-25 07:34:39 on dist-test-slave-57fn"
server_key: "0f7d92be45828144d03a83d3bdd776c2"
server_key_iv: "ad3f2f85017e6d44361c5aa2b55e752b"
server_key_version: "encryptionkey@0"
I20260925 07:34:39.870105 31793 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790321392278483-9480-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790321392278483-9480-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790321392278483-9480-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:34:39.876569 31786 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.66.61" port: 42749 } attempt: 2
I20260925 07:34:39.876919 31788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:39.876806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.61:42749 (local address 127.0.0.1:47606)
0925 07:34:39.876858 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0925 07:34:39.876893 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.61:42749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260925 07:34:39.877202 31788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:39.877119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.61:42749 (local address 127.0.0.1:47612)
0925 07:34:39.877159 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0925 07:34:39.877183 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.61:42749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260925 07:34:39.877283 31786 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.66.61:42749: Network error: Client connection negotiation failed: client connection to 127.9.66.61:42749: connect: Connection refused (error 111)
I20260925 07:34:39.882951 31793 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:34:39.885172 31793 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:34:39.885325 31793 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:34:39.888962 31793 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.61:42749
I20260925 07:34:39.888998 31860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.61:42749 every 8 connection(s)
I20260925 07:34:39.889305 31793 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790321392278483-9480-0/minicluster-data/master-1/data/info.pb
I20260925 07:34:39.890266 31861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:34:39.891776 31861 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.66.62" port: 40975 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 07:34:39 dist-test-slave-57fn krb5kdc[31705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321679, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.61@KRBTEST.COM for kudu/127.9.66.62@KRBTEST.COM
I20260925 07:34:39.896812 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 31793
I20260925 07:34:39.896908 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790321392278483-9480-0/minicluster-data/master-1/wal/instance
I20260925 07:34:39.897096 9480 external_mini_cluster.cc:1428] Setting key 2557b8946fa8ab6efa10a9f997fd5ce8
I20260925 07:34:39.900174 31863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:39.892365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:40975 (local address 127.0.0.1:42836)
0925 07:34:39.892544 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0925 07:34:39.892576 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0925 07:34:39.892996 (+ 420us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:34:39.893208 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:34:39.893211 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:34:39.893420 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:34:39.893698 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:39.893703 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:39.894303 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:39.894303 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:39.895038 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:39.895040 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:39.895380 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:39.895381 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:39.895483 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:39.895891 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:34:39.895898 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:34:39.897263 (+ 1365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:34:39.898873 (+ 1610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:34:39.898880 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:34:39.898882 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:34:39.899112 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:34:39.899319 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:34:39.899319 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:34:39.899320 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:34:39.899419 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:34:39.899720 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:34:39.899724 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:34:39.899828 (+ 104us) client_negotiation.cc:769] Sending connection context
0925 07:34:39.899923 (+ 95us) client_negotiation.cc:240] Negotiation successful
0925 07:34:39.900033 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":109,"threads_started":1}
I20260925 07:34:39.900190 31789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:39.892402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42836 (local address 127.9.66.62:40975)
0925 07:34:39.892501 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:39.892503 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:39.892656 (+ 153us) server_negotiation.cc:407] Connection header received
0925 07:34:39.893072 (+ 416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:39.893073 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:39.893084 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:39.893132 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:34:39.893780 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:39.894233 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:39.895115 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:39.895296 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:39.895350 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:39.897354 (+ 2004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:34:39.897356 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:34:39.897356 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:34:39.897364 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:34:39.898778 (+ 1414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:34:39.899199 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:34:39.899201 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:34:39.899201 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:34:39.899260 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:34:39.899511 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:34:39.899511 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:34:39.899512 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:34:39.899655 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:34:39.899725 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:39.899967 (+ 242us) server_negotiation.cc:299] Negotiation successful
0925 07:34:39.900054 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 07:34:39.900875 31861 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.66.61" port: 42749 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.9.66.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.66.60@KRBTEST.COM" created.
Sep 25 07:34:39 dist-test-slave-57fn krb5kdc[31705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321679, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.61@KRBTEST.COM for kudu/127.9.66.61@KRBTEST.COM
I20260925 07:34:39.911194 31863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:39.901156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.61:42749 (local address 127.0.0.1:47628)
0925 07:34:39.901213 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0925 07:34:39.901228 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 07:34:39.901259 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:34:39.901496 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:34:39.901497 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:34:39.901647 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:34:39.901686 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:39.901688 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:39.902579 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:39.902580 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:39.903079 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:39.903081 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:39.903535 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:39.903535 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:39.903595 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:39.904010 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:34:39.904014 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:34:39.907937 (+ 3923us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:34:39.910115 (+ 2178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:34:39.910116 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:34:39.910116 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:34:39.910286 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:34:39.910502 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:34:39.910503 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:34:39.910503 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:34:39.910556 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:34:39.910929 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:34:39.910930 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:34:39.910978 (+ 48us) client_negotiation.cc:769] Sending connection context
0925 07:34:39.911044 (+ 66us) client_negotiation.cc:240] Negotiation successful
0925 07:34:39.911131 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260925 07:34:39.911334 31865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:39.901224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47628 (local address 127.9.66.61:42749)
0925 07:34:39.901369 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:39.901372 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:39.901387 (+ 15us) server_negotiation.cc:407] Connection header received
0925 07:34:39.901403 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:39.901404 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:39.901413 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:39.901496 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:34:39.901761 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:39.902483 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:39.903170 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:39.903451 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:39.903536 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:39.908039 (+ 4503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:34:39.908046 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:34:39.908046 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:34:39.908052 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:34:39.910032 (+ 1980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:34:39.910368 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:34:39.910371 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:34:39.910372 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:34:39.910433 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:34:39.910623 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:34:39.910624 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:34:39.910624 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:34:39.910827 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:34:39.910917 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:39.911122 (+ 205us) server_negotiation.cc:299] Negotiation successful
0925 07:34:39.911210 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":75,"threads_started":1}
I20260925 07:34:39.912000 31861 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.66.60" port: 36157 } has no permanent_uuid. Determining permanent_uuid...
I20260925 07:34:39.912333 31863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:39.912224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.60:36157 (local address 127.0.0.1:41930)
0925 07:34:39.912268 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0925 07:34:39.912308 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.60:36157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260925 07:34:39.912488 31809 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.66.60:36157: connect: Connection refused (error 111)
I20260925 07:34:39.912912 31863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:39.912821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.60:36157 (local address 127.0.0.1:41936)
0925 07:34:39.912865 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0925 07:34:39.912897 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.60:36157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260925 07:34:39.913034 31861 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.66.60:36157: Network error: Client connection negotiation failed: client connection to 127.9.66.60:36157: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.66.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.60.keytab.
Entry for principal kudu/127.9.66.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.60.keytab.
I20260925 07:34:39.922472 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790321392278483-9480-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790321392278483-9480-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790321392278483-9480-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790321392278483-9480-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.66.60:36157
--webserver_interface=127.9.66.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.60.keytab
--principal=kudu/127.9.66.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:42063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.66.62:40975,127.9.66.61:42749,127.9.66.60:36157
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 07:34:39.962304 31861 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.66.60" port: 36157 } attempt: 1
I20260925 07:34:39.962827 31863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:39.962666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.60:36157 (local address 127.0.0.1:41948)
0925 07:34:39.962753 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0925 07:34:39.962796 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.60:36157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260925 07:34:39.963243 31863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:39.963153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.60:36157 (local address 127.0.0.1:41960)
0925 07:34:39.963198 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0925 07:34:39.963224 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.60:36157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260925 07:34:39.963354 31861 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.66.60:36157: Network error: Client connection negotiation failed: client connection to 127.9.66.60:36157: connect: Connection refused (error 111)
I20260925 07:34:39.968420 31786 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.66.61" port: 42749 } attempt: 3
Sep 25 07:34:39 dist-test-slave-57fn krb5kdc[31705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321679, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.62@KRBTEST.COM for kudu/127.9.66.61@KRBTEST.COM
I20260925 07:34:39.975378 31865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:39.968746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47630 (local address 127.9.66.61:42749)
0925 07:34:39.968805 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:39.968807 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:39.968818 (+ 11us) server_negotiation.cc:407] Connection header received
0925 07:34:39.968891 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:39.968892 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:39.968902 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:39.968973 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:34:39.969315 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:39.969941 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:39.970642 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:39.970892 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:39.970968 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:39.973027 (+ 2059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:34:39.973029 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:34:39.973029 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:34:39.973036 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:34:39.974410 (+ 1374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:34:39.974770 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:34:39.974770 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:34:39.974771 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:34:39.974816 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:34:39.974992 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:34:39.974992 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:34:39.974993 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:34:39.975083 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:34:39.975140 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:39.975277 (+ 137us) server_negotiation.cc:299] Negotiation successful
0925 07:34:39.975331 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 07:34:39.975426 31788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:39.968696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.61:42749 (local address 127.0.0.1:47630)
0925 07:34:39.968744 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0925 07:34:39.968758 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 07:34:39.968824 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:34:39.968978 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:34:39.968979 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:34:39.969161 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:34:39.969242 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:39.969245 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:39.970030 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:39.970032 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:39.970549 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:39.970551 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:39.970965 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:39.970966 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:39.971062 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:39.971525 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:34:39.971529 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:34:39.972897 (+ 1368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:34:39.974493 (+ 1596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:34:39.974494 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:34:39.974495 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:34:39.974700 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:34:39.974876 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:34:39.974876 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:34:39.974877 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:34:39.974937 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:34:39.975150 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:34:39.975151 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:34:39.975195 (+ 44us) client_negotiation.cc:769] Sending connection context
0925 07:34:39.975256 (+ 61us) client_negotiation.cc:240] Negotiation successful
0925 07:34:39.975342 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260925 07:34:39.975816 31786 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.66.60" port: 36157 } has no permanent_uuid. Determining permanent_uuid...
I20260925 07:34:39.976135 31788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:39.976040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.60:36157 (local address 127.0.0.1:41968)
0925 07:34:39.976081 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0925 07:34:39.976112 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.60:36157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260925 07:34:39.976541 31788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:39.976446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.60:36157 (local address 127.0.0.1:41982)
0925 07:34:39.976491 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0925 07:34:39.976518 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.60:36157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260925 07:34:39.976644 31786 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.66.60:36157: Network error: Client connection negotiation failed: client connection to 127.9.66.60:36157: connect: Connection refused (error 111)
W20260925 07:34:40.001276 31868 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 07:34:40.001443 31868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:34:40.001459 31868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:34:40.001538 31868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:34:40.002610 31868 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:34:40.002646 31868 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:34:40.002661 31868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:34:40.002674 31868 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:34:40.002722 31868 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:34:40.004144 31868 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:42063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790321392278483-9480-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790321392278483-9480-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.9.66.62:40975,127.9.66.61:42749,127.9.66.60:36157
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.60.keytab
--principal=kudu/127.9.66.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.66.60:36157
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790321392278483-9480-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.9.66.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31868
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790321392278483-9480-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:34:40.004367 31868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:34:40.004613 31868 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 07:34:40.007763 31786 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.66.60" port: 36157 } attempt: 1
I20260925 07:34:40.008180 31788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:40.008042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.60:36157 (local address 127.0.0.1:41992)
0925 07:34:40.008097 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0925 07:34:40.008134 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.60:36157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260925 07:34:40.008597 31788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:40.008510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.60:36157 (local address 127.0.0.1:41998)
0925 07:34:40.008548 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0925 07:34:40.008573 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.60:36157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260925 07:34:40.008697 31786 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.66.60:36157: Network error: Client connection negotiation failed: client connection to 127.9.66.60:36157: connect: Connection refused (error 111)
W20260925 07:34:40.009552 31874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:34:40.009568 31873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:34:40.009717 31868 server_base.cc:1034] running on GCE node
W20260925 07:34:40.009589 31876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:34:40.009997 31868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:34:40.010293 31868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:34:40.011445 31868 hybrid_clock.cc:648] HybridClock initialized: now 1790321680011429 us; error 26 us; skew 500 ppm
Sep 25 07:34:40 dist-test-slave-57fn krb5kdc[31705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321680, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:34:40.013875 31868 init.cc:377] Logged in from keytab as kudu/127.9.66.60@KRBTEST.COM (short username kudu)
I20260925 07:34:40.014276 31868 webserver.cc:460] Webserver started at http://127.9.66.60:34983/ using document root <none> and password file <none>
I20260925 07:34:40.014498 31868 fs_manager.cc:359] Metadata directory not provided
I20260925 07:34:40.014561 31868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:34:40.014701 31868 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:34:40.015713 31868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790321392278483-9480-0/minicluster-data/master-2/data/instance:
uuid: "3f664db1d07a41b695068b641c8bd6b0"
format_stamp: "Formatted at 2026-09-25 07:34:40 on dist-test-slave-57fn"
server_key: "698bb0ffe0fe13f988de35d8dd8cfc4f"
server_key_iv: "a24c38297199b2f42cd013f89316ae2d"
server_key_version: "encryptionkey@0"
I20260925 07:34:40.016109 31868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790321392278483-9480-0/minicluster-data/master-2/wal/instance:
uuid: "3f664db1d07a41b695068b641c8bd6b0"
format_stamp: "Formatted at 2026-09-25 07:34:40 on dist-test-slave-57fn"
server_key: "698bb0ffe0fe13f988de35d8dd8cfc4f"
server_key_iv: "a24c38297199b2f42cd013f89316ae2d"
server_key_version: "encryptionkey@0"
I20260925 07:34:40.017988 31868 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260925 07:34:40.018942 31883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:34:40.019366 31868 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:34:40.019544 31868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790321392278483-9480-0/minicluster-data/master-2/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790321392278483-9480-0/minicluster-data/master-2/wal
uuid: "3f664db1d07a41b695068b641c8bd6b0"
format_stamp: "Formatted at 2026-09-25 07:34:40 on dist-test-slave-57fn"
server_key: "698bb0ffe0fe13f988de35d8dd8cfc4f"
server_key_iv: "a24c38297199b2f42cd013f89316ae2d"
server_key_version: "encryptionkey@0"
I20260925 07:34:40.019632 31868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790321392278483-9480-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790321392278483-9480-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790321392278483-9480-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:34:40.031504 31861 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.66.60" port: 36157 } attempt: 2
I20260925 07:34:40.031985 31863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:40.031874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.60:36157 (local address 127.0.0.1:42012)
0925 07:34:40.031915 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0925 07:34:40.031950 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.60:36157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260925 07:34:40.032326 31863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:40.032249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.60:36157 (local address 127.0.0.1:42022)
0925 07:34:40.032281 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0925 07:34:40.032309 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.60:36157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260925 07:34:40.032542 31861 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.66.60:36157: Network error: Client connection negotiation failed: client connection to 127.9.66.60:36157: connect: Connection refused (error 111)
I20260925 07:34:40.041698 31868 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:34:40.044401 31868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:34:40.044564 31868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:34:40.049392 31868 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.60:36157
I20260925 07:34:40.049446 31935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.60:36157 every 8 connection(s)
I20260925 07:34:40.049824 31868 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790321392278483-9480-0/minicluster-data/master-2/data/info.pb
I20260925 07:34:40.051054 31936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:34:40.053402 31936 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.66.62" port: 40975 } has no permanent_uuid. Determining permanent_uuid...
I20260925 07:34:40.056600 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 31868
I20260925 07:34:40.056718 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790321392278483-9480-0/minicluster-data/master-2/wal/instance
I20260925 07:34:40.056876 9480 external_mini_cluster.cc:1428] Setting key 43a19ad5cad439d3a2f41ff2f7a6d665
Sep 25 07:34:40 dist-test-slave-57fn krb5kdc[31705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321680, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.60@KRBTEST.COM for kudu/127.9.66.62@KRBTEST.COM
Sep 25 07:34:40 dist-test-slave-57fn krb5kdc[31705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321679, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.66.60@KRBTEST.COM
I20260925 07:34:40.063941 31789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:40.054203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42838 (local address 127.9.66.62:40975)
0925 07:34:40.054279 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:40.054282 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:40.054511 (+ 229us) server_negotiation.cc:407] Connection header received
0925 07:34:40.054921 (+ 410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:40.054923 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:40.054935 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:40.055023 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:34:40.055577 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:40.056399 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:40.057849 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:40.058187 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:40.058261 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:40.060980 (+ 2719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:34:40.060982 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:34:40.060982 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:34:40.060990 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:34:40.062564 (+ 1574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:34:40.062966 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:34:40.062967 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:34:40.062967 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:34:40.063048 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:34:40.063321 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:34:40.063322 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:34:40.063322 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:34:40.063442 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:34:40.063519 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:40.063780 (+ 261us) server_negotiation.cc:299] Negotiation successful
0925 07:34:40.063855 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 07:34:40.064136 31938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:40.054203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:40975 (local address 127.0.0.1:42838)
0925 07:34:40.054390 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0925 07:34:40.054433 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 07:34:40.054835 (+ 402us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:34:40.055117 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:34:40.055120 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:34:40.055327 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:34:40.055494 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:40.055498 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:40.056511 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:40.056512 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:40.057787 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:40.057790 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:40.058294 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:40.058295 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:40.058445 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:40.058920 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:34:40.058929 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:34:40.060876 (+ 1947us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:34:40.062655 (+ 1779us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:34:40.062661 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:34:40.062663 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:34:40.062875 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:34:40.063120 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:34:40.063121 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:34:40.063121 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:34:40.063259 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:34:40.063510 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:34:40.063516 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:34:40.063648 (+ 132us) client_negotiation.cc:769] Sending connection context
0925 07:34:40.063746 (+ 98us) client_negotiation.cc:240] Negotiation successful
0925 07:34:40.063881 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":80,"threads_started":1}
I20260925 07:34:40.064667 31936 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.66.61" port: 42749 } has no permanent_uuid. Determining permanent_uuid...
I20260925 07:34:40.067077 31940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:40.057243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42038 (local address 127.9.66.60:36157)
0925 07:34:40.057882 (+ 639us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:40.057885 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:40.057895 (+ 10us) server_negotiation.cc:407] Connection header received
0925 07:34:40.057912 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:40.057913 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:40.057927 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:40.057997 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:34:40.058713 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:40.059394 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:40.060178 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:40.060531 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:40.060606 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:40.064254 (+ 3648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:34:40.064258 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:34:40.064259 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:34:40.064265 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:34:40.065975 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:34:40.066352 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:34:40.066354 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:34:40.066355 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:34:40.066414 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:34:40.066592 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:34:40.066593 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:34:40.066594 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:34:40.066741 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:34:40.066875 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:40.066957 (+ 82us) server_negotiation.cc:299] Negotiation successful
0925 07:34:40.067012 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":620,"thread_start_us":91,"threads_started":1}
Sep 25 07:34:40 dist-test-slave-57fn krb5kdc[31705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321680, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.60@KRBTEST.COM for kudu/127.9.66.61@KRBTEST.COM
I20260925 07:34:40.071934 31938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:40.064918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.61:42749 (local address 127.0.0.1:47640)
0925 07:34:40.064961 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0925 07:34:40.064969 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 07:34:40.065004 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:34:40.065180 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:34:40.065181 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:34:40.065332 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:34:40.065390 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:40.065392 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:40.066084 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:40.066085 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:40.066789 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:40.066791 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:40.067251 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:40.067252 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:40.067315 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:40.067716 (+ 401us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:34:40.067721 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:34:40.068914 (+ 1193us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:34:40.071009 (+ 2095us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:34:40.071010 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:34:40.071010 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:34:40.071163 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:34:40.071353 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:34:40.071354 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:34:40.071355 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:34:40.071415 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:34:40.071661 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:34:40.071662 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:34:40.071698 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 07:34:40.071772 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 07:34:40.071854 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260925 07:34:40.071977 31865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:40.065012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47640 (local address 127.9.66.61:42749)
0925 07:34:40.065055 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:40.065056 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:40.065064 (+ 8us) server_negotiation.cc:407] Connection header received
0925 07:34:40.065088 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:40.065089 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:40.065097 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:40.065176 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:34:40.065490 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:40.065987 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:40.066861 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:40.067190 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:40.067252 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:40.069011 (+ 1759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:34:40.069012 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:34:40.069012 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:34:40.069019 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:34:40.070926 (+ 1907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:34:40.071232 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:34:40.071233 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:34:40.071234 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:34:40.071288 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:34:40.071484 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:34:40.071485 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:34:40.071485 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:34:40.071586 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:34:40.071655 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:40.071808 (+ 153us) server_negotiation.cc:299] Negotiation successful
0925 07:34:40.071873 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
Sep 25 07:34:40 dist-test-slave-57fn krb5kdc[31705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321679, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.66.62@KRBTEST.COM
I20260925 07:34:40.072892 31936 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.66.60" port: 36157 } has no permanent_uuid. Determining permanent_uuid...
I20260925 07:34:40.075616 31789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:40.068332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42844 (local address 127.9.66.62:40975)
0925 07:34:40.068399 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:40.068400 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:40.068408 (+ 8us) server_negotiation.cc:407] Connection header received
0925 07:34:40.068425 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:40.068426 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:40.068434 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:40.068485 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:34:40.068977 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:40.069437 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:40.070318 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:40.070647 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:40.070714 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:40.073032 (+ 2318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:34:40.073033 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:34:40.073033 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:34:40.073039 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:34:40.074509 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:34:40.074825 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:34:40.074826 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:34:40.074826 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:34:40.074881 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:34:40.075071 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:34:40.075072 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:34:40.075072 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:34:40.075191 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:34:40.075285 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:40.075417 (+ 132us) server_negotiation.cc:299] Negotiation successful
0925 07:34:40.075522 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
Sep 25 07:34:40 dist-test-slave-57fn krb5kdc[31705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321680, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.60@KRBTEST.COM for kudu/127.9.66.60@KRBTEST.COM
I20260925 07:34:40.079901 31940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:40.073162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42054 (local address 127.9.66.60:36157)
0925 07:34:40.073191 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:40.073193 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:40.073232 (+ 39us) server_negotiation.cc:407] Connection header received
0925 07:34:40.073293 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:40.073294 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:40.073304 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:40.073370 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:34:40.073694 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:40.074244 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:40.075133 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:40.075414 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:40.075488 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:40.077285 (+ 1797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:34:40.077286 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:34:40.077287 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:34:40.077293 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:34:40.078675 (+ 1382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:34:40.079107 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:34:40.079108 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:34:40.079108 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:34:40.079162 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:34:40.079355 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:34:40.079355 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:34:40.079356 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:34:40.079443 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:34:40.079490 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:40.079698 (+ 208us) server_negotiation.cc:299] Negotiation successful
0925 07:34:40.079811 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":14}
I20260925 07:34:40.079931 31938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:40.073133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.60:36157 (local address 127.0.0.1:42054)
0925 07:34:40.073186 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0925 07:34:40.073197 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 07:34:40.073245 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:34:40.073357 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:34:40.073358 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:34:40.073558 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:34:40.073590 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:40.073591 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:40.074322 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:40.074323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:40.075080 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:40.075082 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:40.075502 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:40.075503 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:40.075550 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:40.075934 (+ 384us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:34:40.075938 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:34:40.077205 (+ 1267us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:34:40.078793 (+ 1588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:34:40.078793 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:34:40.078794 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:34:40.079029 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:34:40.079228 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:34:40.079230 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:34:40.079230 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:34:40.079291 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:34:40.079507 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:34:40.079508 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:34:40.079563 (+ 55us) client_negotiation.cc:769] Sending connection context
0925 07:34:40.079659 (+ 96us) client_negotiation.cc:240] Negotiation successful
0925 07:34:40.079769 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
Sep 25 07:34:40 dist-test-slave-57fn krb5kdc[31705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321679, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.66.61@KRBTEST.COM
I20260925 07:34:40.081676 31936 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f664db1d07a41b695068b641c8bd6b0: Bootstrap starting.
I20260925 07:34:40.082664 31936 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3f664db1d07a41b695068b641c8bd6b0: Neither blocks nor log segments found. Creating new log.
I20260925 07:34:40.082993 31936 log.cc:824] T 00000000000000000000000000000000 P 3f664db1d07a41b695068b641c8bd6b0: Log is configured to *not* fsync() on all Append() calls
I20260925 07:34:40.083736 31936 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f664db1d07a41b695068b641c8bd6b0: No bootstrap required, opened a new log
I20260925 07:34:40.083801 31865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:40.076315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47644 (local address 127.9.66.61:42749)
0925 07:34:40.076369 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:40.076372 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:40.076412 (+ 40us) server_negotiation.cc:407] Connection header received
0925 07:34:40.076480 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:40.076481 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:40.076487 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:40.076526 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:34:40.077057 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:40.077471 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:40.078482 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:40.078667 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:40.078715 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:40.081002 (+ 2287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:34:40.081003 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:34:40.081004 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:34:40.081010 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:34:40.082660 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:34:40.083043 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:34:40.083045 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:34:40.083045 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:34:40.083101 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:34:40.083322 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:34:40.083322 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:34:40.083323 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:34:40.083437 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:34:40.083595 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:40.083631 (+ 36us) server_negotiation.cc:299] Negotiation successful
0925 07:34:40.083697 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 07:34:40.083976 31786 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.66.60" port: 36157 } attempt: 2
I20260925 07:34:40.085201 9480 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 07:34:40.086107 31936 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f664db1d07a41b695068b641c8bd6b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88eae5eabfc64f108cc52d88079bec35" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 40975 } } peers { permanent_uuid: "8293e1e0553246bcb73add0eaed8eddd" member_type: VOTER last_known_addr { host: "127.9.66.61" port: 42749 } } peers { permanent_uuid: "3f664db1d07a41b695068b641c8bd6b0" member_type: VOTER last_known_addr { host: "127.9.66.60" port: 36157 } }
I20260925 07:34:40.086272 31936 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f664db1d07a41b695068b641c8bd6b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f664db1d07a41b695068b641c8bd6b0, State: Initialized, Role: FOLLOWER
I20260925 07:34:40.086374 31936 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f664db1d07a41b695068b641c8bd6b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88eae5eabfc64f108cc52d88079bec35" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 40975 } } peers { permanent_uuid: "8293e1e0553246bcb73add0eaed8eddd" member_type: VOTER last_known_addr { host: "127.9.66.61" port: 42749 } } peers { permanent_uuid: "3f664db1d07a41b695068b641c8bd6b0" member_type: VOTER last_known_addr { host: "127.9.66.60" port: 36157 } }
I20260925 07:34:40.086660 31942 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f664db1d07a41b695068b641c8bd6b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88eae5eabfc64f108cc52d88079bec35" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 40975 } } peers { permanent_uuid: "8293e1e0553246bcb73add0eaed8eddd" member_type: VOTER last_known_addr { host: "127.9.66.61" port: 42749 } } peers { permanent_uuid: "3f664db1d07a41b695068b641c8bd6b0" member_type: VOTER last_known_addr { host: "127.9.66.60" port: 36157 } } }
I20260925 07:34:40.086752 31942 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f664db1d07a41b695068b641c8bd6b0 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 07:34:40.086935 31936 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f664db1d07a41b695068b641c8bd6b0 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:34:40.086278 31760 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "88eae5eabfc64f108cc52d88079bec35"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42844
Sep 25 07:34:40 dist-test-slave-57fn krb5kdc[31705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321679, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.62@KRBTEST.COM for kudu/127.9.66.60@KRBTEST.COM
W20260925 07:34:40.088708 31953 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3f664db1d07a41b695068b641c8bd6b0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 07:34:40.088764 31953 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 07:34:40.091163 31788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:40.084244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.60:36157 (local address 127.0.0.1:42062)
0925 07:34:40.084289 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0925 07:34:40.084299 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 07:34:40.084336 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:34:40.084470 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:34:40.084471 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:34:40.084628 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:34:40.084663 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:40.084665 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:40.085562 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:40.085563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:40.086024 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:40.086025 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:40.086457 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:40.086458 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:40.086519 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:40.086914 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:34:40.086918 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:34:40.088243 (+ 1325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:34:40.090134 (+ 1891us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:34:40.090135 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:34:40.090135 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:34:40.090338 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:34:40.090538 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:34:40.090539 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:34:40.090539 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:34:40.090594 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:34:40.090907 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:34:40.090907 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:34:40.090947 (+ 40us) client_negotiation.cc:769] Sending connection context
0925 07:34:40.091003 (+ 56us) client_negotiation.cc:240] Negotiation successful
0925 07:34:40.091082 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260925 07:34:40.091500 31940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:40.084274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42062 (local address 127.9.66.60:36157)
0925 07:34:40.084313 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:40.084315 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:40.084324 (+ 9us) server_negotiation.cc:407] Connection header received
0925 07:34:40.084401 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:40.084402 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:40.084411 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:40.084454 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:34:40.084739 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:40.085430 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:40.086120 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:40.086399 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:40.086458 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:40.088347 (+ 1889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:34:40.088348 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:34:40.088349 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:34:40.088355 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:34:40.090040 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:34:40.090419 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:34:40.090420 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:34:40.090420 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:34:40.090471 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:34:40.090656 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:34:40.090657 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:34:40.090657 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:34:40.090800 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:34:40.090879 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:40.091351 (+ 472us) server_negotiation.cc:299] Negotiation successful
0925 07:34:40.091419 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 07:34:40.089076 31868 master_runner.cc:186] Error getting master registration for 127.9.66.62:40975: OK, instance_id { permanent_uuid: "88eae5eabfc64f108cc52d88079bec35" instance_seqno: 1790321679743333 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 07:34:40.091851 31868 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 07:34:40.092823 31786 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88eae5eabfc64f108cc52d88079bec35: Bootstrap starting.
I20260925 07:34:40.093556 31786 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 88eae5eabfc64f108cc52d88079bec35: Neither blocks nor log segments found. Creating new log.
I20260925 07:34:40.093848 31786 log.cc:824] T 00000000000000000000000000000000 P 88eae5eabfc64f108cc52d88079bec35: Log is configured to *not* fsync() on all Append() calls
I20260925 07:34:40.094620 31786 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88eae5eabfc64f108cc52d88079bec35: No bootstrap required, opened a new log
I20260925 07:34:40.095983 31786 raft_consensus.cc:348] T 00000000000000000000000000000000 P 88eae5eabfc64f108cc52d88079bec35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88eae5eabfc64f108cc52d88079bec35" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 40975 } } peers { permanent_uuid: "8293e1e0553246bcb73add0eaed8eddd" member_type: VOTER last_known_addr { host: "127.9.66.61" port: 42749 } } peers { permanent_uuid: "3f664db1d07a41b695068b641c8bd6b0" member_type: VOTER last_known_addr { host: "127.9.66.60" port: 36157 } }
I20260925 07:34:40.096114 31786 raft_consensus.cc:729] T 00000000000000000000000000000000 P 88eae5eabfc64f108cc52d88079bec35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88eae5eabfc64f108cc52d88079bec35, State: Initialized, Role: FOLLOWER
I20260925 07:34:40.096199 31786 consensus_queue.cc:260] T 00000000000000000000000000000000 P 88eae5eabfc64f108cc52d88079bec35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88eae5eabfc64f108cc52d88079bec35" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 40975 } } peers { permanent_uuid: "8293e1e0553246bcb73add0eaed8eddd" member_type: VOTER last_known_addr { host: "127.9.66.61" port: 42749 } } peers { permanent_uuid: "3f664db1d07a41b695068b641c8bd6b0" member_type: VOTER last_known_addr { host: "127.9.66.60" port: 36157 } }
I20260925 07:34:40.096490 31786 sys_catalog.cc:564] T 00000000000000000000000000000000 P 88eae5eabfc64f108cc52d88079bec35 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:34:40.096510 31760 raft_consensus.cc:482] T 00000000000000000000000000000000 P 88eae5eabfc64f108cc52d88079bec35 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 07:34:40.096566 31760 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 88eae5eabfc64f108cc52d88079bec35 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:34:40.097239 31760 raft_consensus.cc:504] T 00000000000000000000000000000000 P 88eae5eabfc64f108cc52d88079bec35 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88eae5eabfc64f108cc52d88079bec35" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 40975 } } peers { permanent_uuid: "8293e1e0553246bcb73add0eaed8eddd" member_type: VOTER last_known_addr { host: "127.9.66.61" port: 42749 } } peers { permanent_uuid: "3f664db1d07a41b695068b641c8bd6b0" member_type: VOTER last_known_addr { host: "127.9.66.60" port: 36157 } }
I20260925 07:34:40.097528 31760 leader_election.cc:288] T 00000000000000000000000000000000 P 88eae5eabfc64f108cc52d88079bec35 [CANDIDATE]: Term 1 election: Requested vote from peers 8293e1e0553246bcb73add0eaed8eddd (127.9.66.61:42749), 3f664db1d07a41b695068b641c8bd6b0 (127.9.66.60:36157)
I20260925 07:34:40.097772 31910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88eae5eabfc64f108cc52d88079bec35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "3f664db1d07a41b695068b641c8bd6b0"
I20260925 07:34:40.097889 31910 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f664db1d07a41b695068b641c8bd6b0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:34:40.099090 31910 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3f664db1d07a41b695068b641c8bd6b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88eae5eabfc64f108cc52d88079bec35 in term 1.
I20260925 07:34:40.099325 31734 leader_election.cc:302] T 00000000000000000000000000000000 P 88eae5eabfc64f108cc52d88079bec35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f664db1d07a41b695068b641c8bd6b0, 88eae5eabfc64f108cc52d88079bec35; no voters:
I20260925 07:34:40.101059 31718 master_runner.cc:186] Error getting master registration for 127.9.66.61:42749: OK, instance_id { permanent_uuid: "8293e1e0553246bcb73add0eaed8eddd" instance_seqno: 1790321679887989 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 07:34:40.103751 31718 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 07:34:40.096450 31956 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88eae5eabfc64f108cc52d88079bec35 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88eae5eabfc64f108cc52d88079bec35" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 40975 } } peers { permanent_uuid: "8293e1e0553246bcb73add0eaed8eddd" member_type: VOTER last_known_addr { host: "127.9.66.61" port: 42749 } } peers { permanent_uuid: "3f664db1d07a41b695068b641c8bd6b0" member_type: VOTER last_known_addr { host: "127.9.66.60" port: 36157 } } }
I20260925 07:34:40.103832 31956 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88eae5eabfc64f108cc52d88079bec35 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 07:34:40.103893 31956 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 88eae5eabfc64f108cc52d88079bec35 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:34:40.103943 31956 raft_consensus.cc:686] T 00000000000000000000000000000000 P 88eae5eabfc64f108cc52d88079bec35 [term 1 LEADER]: Becoming Leader. State: Replica: 88eae5eabfc64f108cc52d88079bec35, State: Running, Role: LEADER
I20260925 07:34:40.104020 31956 consensus_queue.cc:237] T 00000000000000000000000000000000 P 88eae5eabfc64f108cc52d88079bec35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88eae5eabfc64f108cc52d88079bec35" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 40975 } } peers { permanent_uuid: "8293e1e0553246bcb73add0eaed8eddd" member_type: VOTER last_known_addr { host: "127.9.66.61" port: 42749 } } peers { permanent_uuid: "3f664db1d07a41b695068b641c8bd6b0" member_type: VOTER last_known_addr { host: "127.9.66.60" port: 36157 } }
I20260925 07:34:40.104403 31956 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88eae5eabfc64f108cc52d88079bec35 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88eae5eabfc64f108cc52d88079bec35. Latest consensus state: current_term: 1 leader_uuid: "88eae5eabfc64f108cc52d88079bec35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88eae5eabfc64f108cc52d88079bec35" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 40975 } } peers { permanent_uuid: "8293e1e0553246bcb73add0eaed8eddd" member_type: VOTER last_known_addr { host: "127.9.66.61" port: 42749 } } peers { permanent_uuid: "3f664db1d07a41b695068b641c8bd6b0" member_type: VOTER last_known_addr { host: "127.9.66.60" port: 36157 } } }
I20260925 07:34:40.104488 31956 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88eae5eabfc64f108cc52d88079bec35 [sys.catalog]: This master's current role is: LEADER
I20260925 07:34:40.107002 31789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:40.099258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42854 (local address 127.9.66.62:40975)
0925 07:34:40.099301 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:40.099303 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:40.099310 (+ 7us) server_negotiation.cc:407] Connection header received
0925 07:34:40.099329 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:40.099330 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:40.099339 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:40.099387 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:34:40.100180 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:40.100898 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:40.101888 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:40.102168 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:40.102228 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:40.103746 (+ 1518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:34:40.103747 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:34:40.103748 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:34:40.103754 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:34:40.105641 (+ 1887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:34:40.105995 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:34:40.105996 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:34:40.105997 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:34:40.106047 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:34:40.106251 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:34:40.106252 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:34:40.106252 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:34:40.106387 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:34:40.106499 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:40.106816 (+ 317us) server_negotiation.cc:299] Negotiation successful
0925 07:34:40.106890 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
W20260925 07:34:40.108151 31975 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 88eae5eabfc64f108cc52d88079bec35: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 07:34:40.108203 31975 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 07:34:40.109105 31865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:40.099138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47656 (local address 127.9.66.61:42749)
0925 07:34:40.099204 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:40.099206 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:40.099577 (+ 371us) server_negotiation.cc:407] Connection header received
0925 07:34:40.099710 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:40.099711 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:40.099733 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:40.099765 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:34:40.100839 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:40.101554 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:40.103023 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:40.103361 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:40.103404 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:40.105847 (+ 2443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:34:40.105848 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:34:40.105849 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:34:40.105855 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:34:40.107896 (+ 2041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:34:40.108277 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:34:40.108278 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:34:40.108279 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:34:40.108331 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:34:40.108541 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:34:40.108542 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:34:40.108542 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:34:40.108673 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:34:40.108755 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:40.108907 (+ 152us) server_negotiation.cc:299] Negotiation successful
0925 07:34:40.108993 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 07:34:40.109690 31977 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 07:34:40.109863 31977 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 07:34:40.111742 31910 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3f664db1d07a41b695068b641c8bd6b0 [term 1 FOLLOWER]: Refusing update from remote peer 88eae5eabfc64f108cc52d88079bec35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:34:40.112187 31956 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 88eae5eabfc64f108cc52d88079bec35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f664db1d07a41b695068b641c8bd6b0" member_type: VOTER last_known_addr { host: "127.9.66.60" port: 36157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:34:40.113160 31940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:40.099317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42064 (local address 127.9.66.60:36157)
0925 07:34:40.099530 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:40.099531 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:40.099540 (+ 9us) server_negotiation.cc:407] Connection header received
0925 07:34:40.099559 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:40.099559 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:40.099568 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:40.099623 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:34:40.105192 (+ 5569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:40.105869 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:40.106828 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:40.107137 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:40.107191 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:40.109130 (+ 1939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:34:40.109131 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:34:40.109132 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:34:40.109138 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:34:40.111065 (+ 1927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:34:40.111410 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:34:40.111411 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:34:40.111411 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:34:40.111490 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:34:40.112617 (+ 1127us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:34:40.112618 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:34:40.112618 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:34:40.112729 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:34:40.112801 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:40.113005 (+ 204us) server_negotiation.cc:299] Negotiation successful
0925 07:34:40.113077 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194}
I20260925 07:34:40.113416 31942 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f664db1d07a41b695068b641c8bd6b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88eae5eabfc64f108cc52d88079bec35. Latest consensus state: current_term: 1 leader_uuid: "88eae5eabfc64f108cc52d88079bec35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88eae5eabfc64f108cc52d88079bec35" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 40975 } } peers { permanent_uuid: "8293e1e0553246bcb73add0eaed8eddd" member_type: VOTER last_known_addr { host: "127.9.66.61" port: 42749 } } peers { permanent_uuid: "3f664db1d07a41b695068b641c8bd6b0" member_type: VOTER last_known_addr { host: "127.9.66.60" port: 36157 } } }
I20260925 07:34:40.113500 31942 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f664db1d07a41b695068b641c8bd6b0 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 07:34:40.114792 31942 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f664db1d07a41b695068b641c8bd6b0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "88eae5eabfc64f108cc52d88079bec35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88eae5eabfc64f108cc52d88079bec35" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 40975 } } peers { permanent_uuid: "8293e1e0553246bcb73add0eaed8eddd" member_type: VOTER last_known_addr { host: "127.9.66.61" port: 42749 } } peers { permanent_uuid: "3f664db1d07a41b695068b641c8bd6b0" member_type: VOTER last_known_addr { host: "127.9.66.60" port: 36157 } } }
I20260925 07:34:40.114766 31956 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88eae5eabfc64f108cc52d88079bec35 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "88eae5eabfc64f108cc52d88079bec35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88eae5eabfc64f108cc52d88079bec35" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 40975 } } peers { permanent_uuid: "8293e1e0553246bcb73add0eaed8eddd" member_type: VOTER last_known_addr { host: "127.9.66.61" port: 42749 } } peers { permanent_uuid: "3f664db1d07a41b695068b641c8bd6b0" member_type: VOTER last_known_addr { host: "127.9.66.60" port: 36157 } } }
I20260925 07:34:40.114854 31942 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f664db1d07a41b695068b641c8bd6b0 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 07:34:40.114854 31956 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88eae5eabfc64f108cc52d88079bec35 [sys.catalog]: This master's current role is: LEADER
I20260925 07:34:40.115248 31979 mvcc.cc:205] Tried to move back new op lower bound from 7333157601734430720 to 7333157601706668032. Current Snapshot: MvccSnapshot[applied={T|T < 7333157601734430720}]
I20260925 07:34:40.116065 31977 catalog_manager.cc:1347] Generated new cluster ID: 6b3d56f9a59e45b8a0df562fc577791a
I20260925 07:34:40.116171 31977 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 07:34:40.123662 31861 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.66.60" port: 36157 } attempt: 3
Sep 25 07:34:40 dist-test-slave-57fn krb5kdc[31705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321679, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.61@KRBTEST.COM for kudu/127.9.66.60@KRBTEST.COM
I20260925 07:34:40.132069 31863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:40.123930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.60:36157 (local address 127.0.0.1:42080)
0925 07:34:40.123997 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0925 07:34:40.124008 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 07:34:40.124045 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:34:40.124197 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:34:40.124198 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:34:40.124359 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:34:40.124398 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:40.124399 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:40.125390 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:40.125390 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:40.126195 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:40.126197 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:40.126724 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:40.126725 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:40.126791 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:40.127406 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:34:40.127412 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:34:40.129009 (+ 1597us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:34:40.130996 (+ 1987us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:34:40.130998 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:34:40.130998 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:34:40.131205 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:34:40.131421 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:34:40.131422 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:34:40.131422 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:34:40.131486 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:34:40.131769 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:34:40.131770 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:34:40.131822 (+ 52us) client_negotiation.cc:769] Sending connection context
0925 07:34:40.131888 (+ 66us) client_negotiation.cc:240] Negotiation successful
0925 07:34:40.131975 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260925 07:34:40.132522 31940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:40.124034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42080 (local address 127.9.66.60:36157)
0925 07:34:40.124079 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:40.124081 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:40.124088 (+ 7us) server_negotiation.cc:407] Connection header received
0925 07:34:40.124107 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:40.124107 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:40.124116 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:40.124159 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:34:40.124472 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:40.125207 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:40.126293 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:40.126575 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:40.126639 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:40.129106 (+ 2467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:34:40.129108 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:34:40.129108 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:34:40.129114 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:34:40.130898 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:34:40.131287 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:34:40.131288 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:34:40.131288 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:34:40.131349 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:34:40.131554 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:34:40.131555 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:34:40.131555 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:34:40.131680 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:34:40.131734 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:40.132332 (+ 598us) server_negotiation.cc:299] Negotiation successful
0925 07:34:40.132425 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 07:34:40.134596 31861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8293e1e0553246bcb73add0eaed8eddd: Bootstrap starting.
I20260925 07:34:40.135273 31861 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8293e1e0553246bcb73add0eaed8eddd: Neither blocks nor log segments found. Creating new log.
I20260925 07:34:40.135536 31861 log.cc:824] T 00000000000000000000000000000000 P 8293e1e0553246bcb73add0eaed8eddd: Log is configured to *not* fsync() on all Append() calls
I20260925 07:34:40.136479 31861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8293e1e0553246bcb73add0eaed8eddd: No bootstrap required, opened a new log
I20260925 07:34:40.138382 31861 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8293e1e0553246bcb73add0eaed8eddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88eae5eabfc64f108cc52d88079bec35" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 40975 } } peers { permanent_uuid: "8293e1e0553246bcb73add0eaed8eddd" member_type: VOTER last_known_addr { host: "127.9.66.61" port: 42749 } } peers { permanent_uuid: "3f664db1d07a41b695068b641c8bd6b0" member_type: VOTER last_known_addr { host: "127.9.66.60" port: 36157 } }
I20260925 07:34:40.138513 31861 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8293e1e0553246bcb73add0eaed8eddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8293e1e0553246bcb73add0eaed8eddd, State: Initialized, Role: FOLLOWER
I20260925 07:34:40.138588 31861 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8293e1e0553246bcb73add0eaed8eddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88eae5eabfc64f108cc52d88079bec35" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 40975 } } peers { permanent_uuid: "8293e1e0553246bcb73add0eaed8eddd" member_type: VOTER last_known_addr { host: "127.9.66.61" port: 42749 } } peers { permanent_uuid: "3f664db1d07a41b695068b641c8bd6b0" member_type: VOTER last_known_addr { host: "127.9.66.60" port: 36157 } }
I20260925 07:34:40.138908 31861 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8293e1e0553246bcb73add0eaed8eddd [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:34:40.138974 31835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "88eae5eabfc64f108cc52d88079bec35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "8293e1e0553246bcb73add0eaed8eddd"
I20260925 07:34:40.139070 31835 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8293e1e0553246bcb73add0eaed8eddd [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:34:40.139904 31835 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8293e1e0553246bcb73add0eaed8eddd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88eae5eabfc64f108cc52d88079bec35 in term 1.
I20260925 07:34:40.140053 31834 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8293e1e0553246bcb73add0eaed8eddd [term 1 FOLLOWER]: Refusing update from remote peer 88eae5eabfc64f108cc52d88079bec35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:34:40.141659 31984 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 88eae5eabfc64f108cc52d88079bec35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8293e1e0553246bcb73add0eaed8eddd" member_type: VOTER last_known_addr { host: "127.9.66.61" port: 42749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:34:40.144662 32001 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8293e1e0553246bcb73add0eaed8eddd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "88eae5eabfc64f108cc52d88079bec35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88eae5eabfc64f108cc52d88079bec35" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 40975 } } peers { permanent_uuid: "8293e1e0553246bcb73add0eaed8eddd" member_type: VOTER last_known_addr { host: "127.9.66.61" port: 42749 } } peers { permanent_uuid: "3f664db1d07a41b695068b641c8bd6b0" member_type: VOTER last_known_addr { host: "127.9.66.60" port: 36157 } } }
I20260925 07:34:40.144742 32001 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8293e1e0553246bcb73add0eaed8eddd [sys.catalog]: This master's current role is: FOLLOWER
I20260925 07:34:40.144846 32001 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8293e1e0553246bcb73add0eaed8eddd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88eae5eabfc64f108cc52d88079bec35. Latest consensus state: current_term: 1 leader_uuid: "88eae5eabfc64f108cc52d88079bec35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88eae5eabfc64f108cc52d88079bec35" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 40975 } } peers { permanent_uuid: "8293e1e0553246bcb73add0eaed8eddd" member_type: VOTER last_known_addr { host: "127.9.66.61" port: 42749 } } peers { permanent_uuid: "3f664db1d07a41b695068b641c8bd6b0" member_type: VOTER last_known_addr { host: "127.9.66.60" port: 36157 } } }
I20260925 07:34:40.144909 32001 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8293e1e0553246bcb73add0eaed8eddd [sys.catalog]: This master's current role is: FOLLOWER
I20260925 07:34:40.144995 32001 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8293e1e0553246bcb73add0eaed8eddd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "88eae5eabfc64f108cc52d88079bec35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88eae5eabfc64f108cc52d88079bec35" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 40975 } } peers { permanent_uuid: "8293e1e0553246bcb73add0eaed8eddd" member_type: VOTER last_known_addr { host: "127.9.66.61" port: 42749 } } peers { permanent_uuid: "3f664db1d07a41b695068b641c8bd6b0" member_type: VOTER last_known_addr { host: "127.9.66.60" port: 36157 } } }
I20260925 07:34:40.145041 32001 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8293e1e0553246bcb73add0eaed8eddd [sys.catalog]: This master's current role is: FOLLOWER
I20260925 07:34:40.145407 31999 mvcc.cc:205] Tried to move back new op lower bound from 7333157601734430720 to 7333157601706668032. Current Snapshot: MvccSnapshot[applied={T|T < 7333157601734430720}]
I20260925 07:34:40.146320 31970 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88eae5eabfc64f108cc52d88079bec35 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "88eae5eabfc64f108cc52d88079bec35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88eae5eabfc64f108cc52d88079bec35" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 40975 } } peers { permanent_uuid: "8293e1e0553246bcb73add0eaed8eddd" member_type: VOTER last_known_addr { host: "127.9.66.61" port: 42749 } } peers { permanent_uuid: "3f664db1d07a41b695068b641c8bd6b0" member_type: VOTER last_known_addr { host: "127.9.66.60" port: 36157 } } }
I20260925 07:34:40.146413 31970 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88eae5eabfc64f108cc52d88079bec35 [sys.catalog]: This master's current role is: LEADER
I20260925 07:34:40.147667 31977 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 07:34:40.148430 31977 catalog_manager.cc:1504] Loading token signing keys...
W20260925 07:34:40.150081 31997 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8293e1e0553246bcb73add0eaed8eddd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 07:34:40.150136 31997 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 07:34:40.169608 31977 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 88eae5eabfc64f108cc52d88079bec35: Generated new TSK 0
I20260925 07:34:40.169839 31977 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 07:34:40.191077 31865 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:34:40.189539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47662 (local address 127.9.66.61:42749)
0925 07:34:40.189589 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:40.189592 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:40.190416 (+ 824us) server_negotiation.cc:407] Connection header received
0925 07:34:40.190536 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:40.190538 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:40.190549 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:40.190579 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:34:40.191005 (+ 426us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47662: BlockingRecv error: recv got EOF from 127.0.0.1:47662 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260925 07:34:40.191974 31789 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:34:40.189912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42870 (local address 127.9.66.62:40975)
0925 07:34:40.189952 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:40.189954 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:40.191484 (+ 1530us) server_negotiation.cc:407] Connection header received
0925 07:34:40.191583 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:40.191584 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:40.191594 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:40.191622 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:34:40.191912 (+ 290us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42870: BlockingRecv error: recv got EOF from 127.0.0.1:42870 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260925 07:34:40.192772 31940 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:34:40.190188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42084 (local address 127.9.66.60:36157)
0925 07:34:40.190225 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:40.190227 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:40.192295 (+ 2068us) server_negotiation.cc:407] Connection header received
0925 07:34:40.192396 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:40.192397 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:40.192407 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:40.192434 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:34:40.192718 (+ 284us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42084: BlockingRecv error: recv got EOF from 127.0.0.1:42084 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260925 07:34:40.194996 31865 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:34:40.194347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47674 (local address 127.9.66.61:42749)
0925 07:34:40.194378 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:40.194380 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:40.194552 (+ 172us) server_negotiation.cc:407] Connection header received
0925 07:34:40.194642 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:40.194643 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:40.194658 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:40.194684 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:34:40.194954 (+ 270us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47674: BlockingRecv error: recv got EOF from 127.0.0.1:47674 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260925 07:34:40.197969 31865 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:34:40.197191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47688 (local address 127.9.66.61:42749)
0925 07:34:40.197219 (+ 28us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:40.197220 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:40.197471 (+ 251us) server_negotiation.cc:407] Connection header received
0925 07:34:40.197610 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:40.197611 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:40.197621 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:40.197648 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:34:40.197932 (+ 284us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47688: BlockingRecv error: recv got EOF from 127.0.0.1:47688 (error 108)
Metrics: {"server-negotiator.queue_time_us":14}
W20260925 07:34:40.201802 31865 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:34:40.201069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47694 (local address 127.9.66.61:42749)
0925 07:34:40.201101 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:40.201103 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:40.201299 (+ 196us) server_negotiation.cc:407] Connection header received
0925 07:34:40.201382 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:40.201383 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:40.201392 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:40.201425 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:34:40.201767 (+ 342us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47694: BlockingRecv error: recv got EOF from 127.0.0.1:47694 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260925 07:34:40.207690 31865 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:34:40.206915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47698 (local address 127.9.66.61:42749)
0925 07:34:40.206945 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:40.206947 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:40.207204 (+ 257us) server_negotiation.cc:407] Connection header received
0925 07:34:40.207297 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:40.207298 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:40.207307 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:40.207336 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:34:40.207652 (+ 316us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47698: BlockingRecv error: recv got EOF from 127.0.0.1:47698 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260925 07:34:40.217799 31865 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:34:40.216935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47710 (local address 127.9.66.61:42749)
0925 07:34:40.216970 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:40.216972 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:40.217200 (+ 228us) server_negotiation.cc:407] Connection header received
0925 07:34:40.217293 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:40.217294 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:40.217304 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:40.217331 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:34:40.217758 (+ 427us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47710: BlockingRecv error: recv got EOF from 127.0.0.1:47710 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260925 07:34:40.236024 31865 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:34:40.235112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47714 (local address 127.9.66.61:42749)
0925 07:34:40.235152 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:40.235154 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:40.235403 (+ 249us) server_negotiation.cc:407] Connection header received
0925 07:34:40.235520 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:40.235521 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:40.235532 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:40.235561 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:34:40.235968 (+ 407us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47714: BlockingRecv error: recv got EOF from 127.0.0.1:47714 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260925 07:34:40.270457 31865 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:34:40.269545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47720 (local address 127.9.66.61:42749)
0925 07:34:40.269601 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:40.269603 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:40.269883 (+ 280us) server_negotiation.cc:407] Connection header received
0925 07:34:40.269983 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:40.269984 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:40.269996 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:40.270026 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:34:40.270382 (+ 356us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47720: BlockingRecv error: recv got EOF from 127.0.0.1:47720 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
W20260925 07:34:40.336673 31865 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:34:40.335964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47732 (local address 127.9.66.61:42749)
0925 07:34:40.336027 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:40.336029 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:40.336130 (+ 101us) server_negotiation.cc:407] Connection header received
0925 07:34:40.336200 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:40.336201 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:40.336211 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:40.336257 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:34:40.336577 (+ 320us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47732: BlockingRecv error: recv got EOF from 127.0.0.1:47732 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260925 07:34:40.466804 31865 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:34:40.466019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47736 (local address 127.9.66.61:42749)
0925 07:34:40.466100 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:40.466103 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:40.466201 (+ 98us) server_negotiation.cc:407] Connection header received
0925 07:34:40.466297 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:40.466298 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:40.466311 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:40.466414 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:34:40.466717 (+ 303us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47736: BlockingRecv error: recv got EOF from 127.0.0.1:47736 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260925 07:34:40.724784 31865 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:34:40.724122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47742 (local address 127.9.66.61:42749)
0925 07:34:40.724174 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:40.724176 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:40.724188 (+ 12us) server_negotiation.cc:407] Connection header received
0925 07:34:40.724278 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:40.724279 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:40.724291 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:40.724362 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:34:40.724697 (+ 335us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47742: BlockingRecv error: recv got EOF from 127.0.0.1:47742 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 07:34:41.089170 31953 catalog_manager.cc:1259] Loaded cluster ID: 6b3d56f9a59e45b8a0df562fc577791a
I20260925 07:34:41.089242 31953 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3f664db1d07a41b695068b641c8bd6b0: loading cluster ID for follower catalog manager: success
I20260925 07:34:41.090061 31953 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3f664db1d07a41b695068b641c8bd6b0: acquiring CA information for follower catalog manager: success
I20260925 07:34:41.090312 31953 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3f664db1d07a41b695068b641c8bd6b0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 07:34:41.150517 31997 catalog_manager.cc:1259] Loaded cluster ID: 6b3d56f9a59e45b8a0df562fc577791a
I20260925 07:34:41.150625 31997 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8293e1e0553246bcb73add0eaed8eddd: loading cluster ID for follower catalog manager: success
I20260925 07:34:41.151471 31997 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8293e1e0553246bcb73add0eaed8eddd: acquiring CA information for follower catalog manager: success
I20260925 07:34:41.151799 31997 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8293e1e0553246bcb73add0eaed8eddd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 07:34:41.240710 32069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:41.237960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47756 (local address 127.9.66.61:42749)
0925 07:34:41.238124 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:41.238126 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:41.238139 (+ 13us) server_negotiation.cc:407] Connection header received
0925 07:34:41.238243 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:41.238244 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:41.238294 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:41.238383 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 07:34:41.238560 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:41.239069 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:41.239880 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:41.240145 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:41.240205 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:41.240355 (+ 150us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 07:34:41.240468 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 07:34:41.240530 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:41.240624 (+ 94us) server_negotiation.cc:299] Negotiation successful
0925 07:34:41.240645 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":111,"threads_started":1}
I20260925 07:34:41.245007 32075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:41.242659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42100 (local address 127.9.66.60:36157)
0925 07:34:41.242748 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:41.242749 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:41.242757 (+ 8us) server_negotiation.cc:407] Connection header received
0925 07:34:41.242815 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:41.242816 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:41.242847 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:41.242916 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 07:34:41.243038 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:41.243517 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:41.244183 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:41.244428 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:41.244484 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:41.244645 (+ 161us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 07:34:41.244750 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 07:34:41.244802 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:41.244868 (+ 66us) server_negotiation.cc:299] Negotiation successful
0925 07:34:41.244886 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73,"thread_start_us":46,"threads_started":1}
I20260925 07:34:41.249670 32082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:41.246716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42874 (local address 127.9.66.62:40975)
0925 07:34:41.246822 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:41.246824 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:41.246834 (+ 10us) server_negotiation.cc:407] Connection header received
0925 07:34:41.246881 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:41.246882 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:41.246931 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:41.247012 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 07:34:41.247165 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:41.247974 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:41.248750 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:41.248969 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:41.249045 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:41.249202 (+ 157us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 07:34:41.249280 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 07:34:41.249327 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:41.249556 (+ 229us) server_negotiation.cc:299] Negotiation successful
0925 07:34:41.249583 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85,"thread_start_us":51,"threads_started":1}
I20260925 07:34:41.250036 32075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:41.246857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42102 (local address 127.9.66.60:36157)
0925 07:34:41.246891 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:41.246892 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:41.247170 (+ 278us) server_negotiation.cc:407] Connection header received
0925 07:34:41.247282 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:41.247283 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:41.247292 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:41.247339 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 07:34:41.247452 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:41.247960 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:41.249099 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:41.249344 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:41.249419 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:41.249568 (+ 149us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 07:34:41.249598 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 07:34:41.249650 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:41.249939 (+ 289us) server_negotiation.cc:299] Negotiation successful
0925 07:34:41.249966 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 07:34:41.250276 32069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:41.246660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47762 (local address 127.9.66.61:42749)
0925 07:34:41.246707 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:41.246709 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:41.246904 (+ 195us) server_negotiation.cc:407] Connection header received
0925 07:34:41.247015 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:41.247015 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:41.247025 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:41.247078 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 07:34:41.247227 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:41.247956 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:41.249447 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:41.249744 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:41.249806 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:41.249994 (+ 188us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 07:34:41.250023 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 07:34:41.250050 (+ 27us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:41.250165 (+ 115us) server_negotiation.cc:299] Negotiation successful
0925 07:34:41.250187 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 07:34:41.251813 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 31718
I20260925 07:34:41.257688 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 31793
I20260925 07:34:41.262543 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 31868
2026-09-25T07:34:41Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2757 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790321392278483-9480-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 07:34:41 dist-test-slave-57fn krb5kdc[32086](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 07:34:41 dist-test-slave-57fn krb5kdc[32086](info): set up 2 sockets
Sep 25 07:34:41 dist-test-slave-57fn krb5kdc[32086](info): commencing operation
krb5kdc: starting...
W20260925 07:34:42.310081 9480 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790321392278483-9480-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790321392278483-9480-0/krb5kdc/test-user.keytab.
Sep 25 07:34:42 dist-test-slave-57fn krb5kdc[32086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321682, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T07:34:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T07:34:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.66.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.66.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.66.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab.
Entry for principal kudu/127.9.66.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab.
I20260925 07:34:42.424221 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790321392278483-9480-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790321392278483-9480-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790321392278483-9480-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.66.62:41807
--webserver_interface=127.9.66.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab
--principal=kudu/127.9.66.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:42415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.66.62:41807,127.9.66.61:45751,127.9.66.60:34759
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 07:34:42.506119 32099 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 07:34:42.506367 32099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:34:42.506385 32099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:34:42.506443 32099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:34:42.507588 32099 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:34:42.507629 32099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:34:42.507643 32099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:34:42.507656 32099 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:34:42.507701 32099 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:34:42.509238 32099 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:42415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790321392278483-9480-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790321392278483-9480-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.66.62:41807,127.9.66.61:45751,127.9.66.60:34759
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.62.keytab
--principal=kudu/127.9.66.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.66.62:41807
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.66.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32099
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790321392278483-9480-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:34:42.509492 32099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:34:42.509766 32099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:34:42.514493 32105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:34:42.514549 32104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:34:42.514521 32107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:34:42.514854 32099 server_base.cc:1034] running on GCE node
I20260925 07:34:42.515120 32099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:34:42.515442 32099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:34:42.516644 32099 hybrid_clock.cc:648] HybridClock initialized: now 1790321682516633 us; error 43 us; skew 500 ppm
Sep 25 07:34:42 dist-test-slave-57fn krb5kdc[32086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321682, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:34:42.519250 32099 init.cc:377] Logged in from keytab as kudu/127.9.66.62@KRBTEST.COM (short username kudu)
I20260925 07:34:42.519739 32099 webserver.cc:460] Webserver started at http://127.9.66.62:46185/ using document root <none> and password file <none>
I20260925 07:34:42.519968 32099 fs_manager.cc:359] Metadata directory not provided
I20260925 07:34:42.520033 32099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:34:42.520182 32099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:34:42.521229 32099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790321392278483-9480-0/minicluster-data/master-0/data/instance:
uuid: "c9618ba2b332490ebc7f55e7bf191dce"
format_stamp: "Formatted at 2026-09-25 07:34:42 on dist-test-slave-57fn"
server_key: "6e6f27ddafa179575113c66f551cd2c7"
server_key_iv: "15e571212f42f212977929e16e759b24"
server_key_version: "encryptionkey@0"
I20260925 07:34:42.521659 32099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790321392278483-9480-0/minicluster-data/master-0/wal/instance:
uuid: "c9618ba2b332490ebc7f55e7bf191dce"
format_stamp: "Formatted at 2026-09-25 07:34:42 on dist-test-slave-57fn"
server_key: "6e6f27ddafa179575113c66f551cd2c7"
server_key_iv: "15e571212f42f212977929e16e759b24"
server_key_version: "encryptionkey@0"
I20260925 07:34:42.523029 32099 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 07:34:42.523828 32114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:34:42.524003 32099 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 07:34:42.524101 32099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790321392278483-9480-0/minicluster-data/master-0/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790321392278483-9480-0/minicluster-data/master-0/wal
uuid: "c9618ba2b332490ebc7f55e7bf191dce"
format_stamp: "Formatted at 2026-09-25 07:34:42 on dist-test-slave-57fn"
server_key: "6e6f27ddafa179575113c66f551cd2c7"
server_key_iv: "15e571212f42f212977929e16e759b24"
server_key_version: "encryptionkey@0"
I20260925 07:34:42.524202 32099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790321392278483-9480-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790321392278483-9480-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790321392278483-9480-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:34:42.532346 32099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:34:42.534217 32099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:34:42.534404 32099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:34:42.538594 32099 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.62:41807
I20260925 07:34:42.538625 32166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.62:41807 every 8 connection(s)
I20260925 07:34:42.538992 32099 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790321392278483-9480-0/minicluster-data/master-0/data/info.pb
I20260925 07:34:42.539935 32167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:34:42.541440 32167 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.66.62" port: 41807 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 07:34:42 dist-test-slave-57fn krb5kdc[32086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321682, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.62@KRBTEST.COM for kudu/127.9.66.62@KRBTEST.COM
I20260925 07:34:42.548794 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 32099
I20260925 07:34:42.548918 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790321392278483-9480-0/minicluster-data/master-0/wal/instance
I20260925 07:34:42.549137 9480 external_mini_cluster.cc:1428] Setting key 44450df7858b537d7b39ec457f36f8ed
I20260925 07:34:42.551556 32170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:42.542244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45430 (local address 127.9.66.62:41807)
0925 07:34:42.542399 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:42.542407 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:42.542542 (+ 135us) server_negotiation.cc:407] Connection header received
0925 07:34:42.543043 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:42.543045 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:42.543061 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:42.543139 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:34:42.543755 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:42.544574 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:42.545416 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:42.545794 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:42.545872 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:42.547829 (+ 1957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:34:42.547835 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:34:42.547836 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:34:42.547842 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:34:42.549775 (+ 1933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:34:42.550151 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:34:42.550154 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:34:42.550154 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:34:42.550267 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:34:42.550555 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:34:42.550555 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:34:42.550556 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:34:42.550870 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:34:42.550957 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:42.551251 (+ 294us) server_negotiation.cc:299] Negotiation successful
0925 07:34:42.551385 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":90,"threads_started":1}
I20260925 07:34:42.551587 32169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:42.542189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:41807 (local address 127.0.0.1:45430)
0925 07:34:42.542410 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0925 07:34:42.542447 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 07:34:42.542923 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:34:42.543211 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:34:42.543214 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:34:42.543525 (+ 311us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:34:42.543670 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:42.543674 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:42.544667 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:42.544668 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:42.545341 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:42.545343 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:42.545884 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:42.545885 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:42.545941 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:42.546416 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:34:42.546423 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:34:42.547754 (+ 1331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:34:42.549856 (+ 2102us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:34:42.549862 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:34:42.549864 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:34:42.550064 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:34:42.550369 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:34:42.550370 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:34:42.550370 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:34:42.550457 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:34:42.550979 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:34:42.550985 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:34:42.551067 (+ 82us) client_negotiation.cc:769] Sending connection context
0925 07:34:42.551178 (+ 111us) client_negotiation.cc:240] Negotiation successful
0925 07:34:42.551474 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"mutex_wait_us":98,"thread_start_us":102,"threads_started":1}
I20260925 07:34:42.552364 32167 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.66.61" port: 45751 } has no permanent_uuid. Determining permanent_uuid...
I20260925 07:34:42.552749 32169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:42.552615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.61:45751 (local address 127.0.0.1:43548)
0925 07:34:42.552672 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0925 07:34:42.552720 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.61:45751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260925 07:34:42.552886 32117 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.66.61:45751: connect: Connection refused (error 111)
I20260925 07:34:42.553323 32169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:42.553219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.61:45751 (local address 127.0.0.1:43560)
0925 07:34:42.553268 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0925 07:34:42.553295 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.61:45751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260925 07:34:42.553484 32167 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.66.61:45751: Network error: Client connection negotiation failed: client connection to 127.9.66.61:45751: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.9.66.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.66.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.66.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.61.keytab.
Entry for principal kudu/127.9.66.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.61.keytab.
I20260925 07:34:42.574381 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790321392278483-9480-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790321392278483-9480-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790321392278483-9480-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790321392278483-9480-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.66.61:45751
--webserver_interface=127.9.66.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.61.keytab
--principal=kudu/127.9.66.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:42415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.66.62:41807,127.9.66.61:45751,127.9.66.60:34759
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 07:34:42.602739 32167 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.66.61" port: 45751 } attempt: 1
I20260925 07:34:42.603251 32169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:42.603108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.61:45751 (local address 127.0.0.1:43566)
0925 07:34:42.603171 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0925 07:34:42.603217 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.61:45751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260925 07:34:42.603597 32169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:42.603490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.61:45751 (local address 127.0.0.1:43576)
0925 07:34:42.603542 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0925 07:34:42.603573 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.61:45751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260925 07:34:42.603710 32167 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.66.61:45751: Network error: Client connection negotiation failed: client connection to 127.9.66.61:45751: connect: Connection refused (error 111)
W20260925 07:34:42.654830 32174 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 07:34:42.655104 32174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:34:42.655136 32174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:34:42.655191 32174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:34:42.656318 32174 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:34:42.656387 32174 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:34:42.656415 32174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:34:42.656447 32174 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:34:42.656481 32174 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:34:42.657997 32174 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:42415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790321392278483-9480-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790321392278483-9480-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.9.66.62:41807,127.9.66.61:45751,127.9.66.60:34759
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.61.keytab
--principal=kudu/127.9.66.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.66.61:45751
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790321392278483-9480-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.9.66.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32174
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790321392278483-9480-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:34:42.658279 32174 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:34:42.658521 32174 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:34:42.663345 32180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:34:42.663425 32179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:34:42.663359 32182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:34:42.663566 32174 server_base.cc:1034] running on GCE node
I20260925 07:34:42.663779 32174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:34:42.664011 32174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:34:42.665189 32174 hybrid_clock.cc:648] HybridClock initialized: now 1790321682665169 us; error 37 us; skew 500 ppm
Sep 25 07:34:42 dist-test-slave-57fn krb5kdc[32086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321682, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:34:42.667843 32174 init.cc:377] Logged in from keytab as kudu/127.9.66.61@KRBTEST.COM (short username kudu)
I20260925 07:34:42.668300 32174 webserver.cc:460] Webserver started at http://127.9.66.61:42015/ using document root <none> and password file <none>
I20260925 07:34:42.668565 32174 fs_manager.cc:359] Metadata directory not provided
I20260925 07:34:42.668609 32174 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:34:42.668766 32174 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:34:42.669855 32174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790321392278483-9480-0/minicluster-data/master-1/data/instance:
uuid: "87b96a210f44439e9b395de6d3eb236d"
format_stamp: "Formatted at 2026-09-25 07:34:42 on dist-test-slave-57fn"
server_key: "4bbebaae2c447a7e53c390f837caa53d"
server_key_iv: "d3263a7d886379b3378804d55fb7b258"
server_key_version: "encryptionkey@0"
I20260925 07:34:42.670208 32174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790321392278483-9480-0/minicluster-data/master-1/wal/instance:
uuid: "87b96a210f44439e9b395de6d3eb236d"
format_stamp: "Formatted at 2026-09-25 07:34:42 on dist-test-slave-57fn"
server_key: "4bbebaae2c447a7e53c390f837caa53d"
server_key_iv: "d3263a7d886379b3378804d55fb7b258"
server_key_version: "encryptionkey@0"
I20260925 07:34:42.671491 32174 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:34:42.671860 32167 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.66.61" port: 45751 } attempt: 2
I20260925 07:34:42.672240 32189 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:34:42.672271 32169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:42.672140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.61:45751 (local address 127.0.0.1:43588)
0925 07:34:42.672198 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0925 07:34:42.672245 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.61:45751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260925 07:34:42.672413 32174 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:34:42.672545 32174 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790321392278483-9480-0/minicluster-data/master-1/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790321392278483-9480-0/minicluster-data/master-1/wal
uuid: "87b96a210f44439e9b395de6d3eb236d"
format_stamp: "Formatted at 2026-09-25 07:34:42 on dist-test-slave-57fn"
server_key: "4bbebaae2c447a7e53c390f837caa53d"
server_key_iv: "d3263a7d886379b3378804d55fb7b258"
server_key_version: "encryptionkey@0"
I20260925 07:34:42.672634 32174 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790321392278483-9480-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790321392278483-9480-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790321392278483-9480-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:34:42.672681 32169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:42.672573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.61:45751 (local address 127.0.0.1:43600)
0925 07:34:42.672633 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0925 07:34:42.672657 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.61:45751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260925 07:34:42.672828 32167 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.66.61:45751: Network error: Client connection negotiation failed: client connection to 127.9.66.61:45751: connect: Connection refused (error 111)
I20260925 07:34:42.691241 32174 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:34:42.693317 32174 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:34:42.693462 32174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:34:42.696887 32174 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.61:45751
I20260925 07:34:42.696942 32241 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.61:45751 every 8 connection(s)
I20260925 07:34:42.697276 32174 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790321392278483-9480-0/minicluster-data/master-1/data/info.pb
I20260925 07:34:42.698354 32242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:34:42.698788 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 32174
I20260925 07:34:42.698879 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790321392278483-9480-0/minicluster-data/master-1/wal/instance
I20260925 07:34:42.699086 9480 external_mini_cluster.cc:1428] Setting key 61949084066e505479e9bad21de08f17
I20260925 07:34:42.700012 32242 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.66.62" port: 41807 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.9.66.60@KRBTEST.COM; defaulting to no policy
Sep 25 07:34:42 dist-test-slave-57fn krb5kdc[32086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321682, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.61@KRBTEST.COM for kudu/127.9.66.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.66.60@KRBTEST.COM" created.
I20260925 07:34:42.709139 32245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:42.700740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:41807 (local address 127.0.0.1:45446)
0925 07:34:42.700934 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0925 07:34:42.700980 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 07:34:42.701593 (+ 613us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:34:42.701862 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:34:42.701865 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:34:42.702126 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:34:42.702339 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:42.702345 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:42.703021 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:42.703022 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:42.703800 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:42.703802 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:42.704213 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:42.704215 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:42.704335 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:42.704810 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:34:42.704818 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:34:42.706146 (+ 1328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:34:42.707663 (+ 1517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:34:42.707670 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:34:42.707671 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:34:42.707918 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:34:42.708151 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:34:42.708152 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:34:42.708152 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:34:42.708268 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:34:42.708669 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:34:42.708676 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:34:42.708783 (+ 107us) client_negotiation.cc:769] Sending connection context
0925 07:34:42.708869 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 07:34:42.708992 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":94,"threads_started":1}
I20260925 07:34:42.709173 32170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:42.700761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45446 (local address 127.9.66.62:41807)
0925 07:34:42.700827 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:42.700830 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:42.701071 (+ 241us) server_negotiation.cc:407] Connection header received
0925 07:34:42.701705 (+ 634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:42.701706 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:42.701720 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:42.701780 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:34:42.702431 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:42.702918 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:42.703901 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:42.704114 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:42.704192 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:42.706235 (+ 2043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:34:42.706237 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:34:42.706237 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:34:42.706243 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:34:42.707602 (+ 1359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:34:42.707990 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:34:42.707990 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:34:42.707991 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:34:42.708068 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:34:42.708410 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:34:42.708411 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:34:42.708411 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:34:42.708579 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:34:42.708673 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:42.708921 (+ 248us) server_negotiation.cc:299] Negotiation successful
0925 07:34:42.709025 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 07:34:42.709717 32242 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.66.61" port: 45751 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 07:34:42 dist-test-slave-57fn krb5kdc[32086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321682, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.61@KRBTEST.COM for kudu/127.9.66.61@KRBTEST.COM
I20260925 07:34:42.717857 32245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:42.709955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.61:45751 (local address 127.0.0.1:43606)
0925 07:34:42.710001 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0925 07:34:42.710011 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 07:34:42.710051 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:34:42.710367 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:34:42.710368 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:34:42.710480 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:34:42.710522 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:42.710523 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:42.711856 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:42.711856 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:42.712330 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:42.712332 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:42.713026 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:42.713026 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:42.713075 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:42.713541 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:34:42.713547 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:34:42.714972 (+ 1425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:34:42.716642 (+ 1670us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:34:42.716643 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:34:42.716643 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:34:42.716888 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:34:42.717096 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:34:42.717097 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:34:42.717097 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:34:42.717155 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:34:42.717468 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:34:42.717468 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:34:42.717504 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 07:34:42.717646 (+ 142us) client_negotiation.cc:240] Negotiation successful
0925 07:34:42.717752 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260925 07:34:42.718029 32247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:42.710009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43606 (local address 127.9.66.61:45751)
0925 07:34:42.710187 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:42.710193 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:42.710218 (+ 25us) server_negotiation.cc:407] Connection header received
0925 07:34:42.710245 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:42.710247 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:42.710263 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:42.710348 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:34:42.710597 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:42.711773 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:42.712411 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:42.712940 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:42.713033 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:42.715060 (+ 2027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:34:42.715066 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:34:42.715067 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:34:42.715072 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:34:42.716555 (+ 1483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:34:42.716979 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:34:42.716981 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:34:42.716981 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:34:42.717028 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:34:42.717215 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:34:42.717215 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:34:42.717215 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:34:42.717345 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:34:42.717405 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:42.717699 (+ 294us) server_negotiation.cc:299] Negotiation successful
0925 07:34:42.717844 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":52,"server-negotiator.queue_time_us":144,"thread_start_us":94,"threads_started":1}
I20260925 07:34:42.718582 32242 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.66.60" port: 34759 } has no permanent_uuid. Determining permanent_uuid...
I20260925 07:34:42.718950 32245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:42.718811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.60:34759 (local address 127.0.0.1:59094)
0925 07:34:42.718859 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0925 07:34:42.718915 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.60:34759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260925 07:34:42.719074 32192 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.66.60:34759: connect: Connection refused (error 111)
I20260925 07:34:42.719521 32245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:42.719418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.60:34759 (local address 127.0.0.1:59106)
0925 07:34:42.719466 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0925 07:34:42.719501 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.60:34759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260925 07:34:42.719695 32242 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.66.60:34759: Network error: Client connection negotiation failed: client connection to 127.9.66.60:34759: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.66.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.60.keytab.
Entry for principal kudu/127.9.66.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.60.keytab.
I20260925 07:34:42.724939 9480 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1eZXJS/build/release/bin/kudu
/tmp/dist-test-task1eZXJS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790321392278483-9480-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790321392278483-9480-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790321392278483-9480-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790321392278483-9480-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.66.60:34759
--webserver_interface=127.9.66.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.60.keytab
--principal=kudu/127.9.66.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.66.20:42415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.66.62:41807,127.9.66.61:45751,127.9.66.60:34759
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790321392278483-9480-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790321392278483-9480-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790321392278483-9480-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 07:34:42.764004 32167 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.66.61" port: 45751 } attempt: 3
Sep 25 07:34:42 dist-test-slave-57fn krb5kdc[32086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321682, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.62@KRBTEST.COM for kudu/127.9.66.61@KRBTEST.COM
I20260925 07:34:42.768900 32242 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.66.60" port: 34759 } attempt: 1
I20260925 07:34:42.769308 32245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:42.769151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.60:34759 (local address 127.0.0.1:59108)
0925 07:34:42.769223 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0925 07:34:42.769255 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.60:34759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260925 07:34:42.769830 32250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:42.769569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.60:34759 (local address 127.0.0.1:59124)
0925 07:34:42.769714 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 07:34:42.769772 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.60:34759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":98,"threads_started":1}
W20260925 07:34:42.770531 32242 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.66.60:34759: Network error: Client connection negotiation failed: client connection to 127.9.66.60:34759: connect: Connection refused (error 111)
I20260925 07:34:42.771561 32247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:42.764597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43618 (local address 127.9.66.61:45751)
0925 07:34:42.764676 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:42.764678 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:42.764690 (+ 12us) server_negotiation.cc:407] Connection header received
0925 07:34:42.764768 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:42.764769 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:42.764785 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:42.764846 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:34:42.765180 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:42.765746 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:42.766465 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:42.766673 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:42.766753 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:42.768806 (+ 2053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:34:42.768807 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:34:42.768808 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:34:42.768815 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:34:42.770368 (+ 1553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:34:42.770796 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:34:42.770797 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:34:42.770797 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:34:42.770857 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:34:42.771064 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:34:42.771065 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:34:42.771065 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:34:42.771176 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:34:42.771251 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:42.771399 (+ 148us) server_negotiation.cc:299] Negotiation successful
0925 07:34:42.771479 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 07:34:42.771569 32169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:42.764516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.61:45751 (local address 127.0.0.1:43618)
0925 07:34:42.764586 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0925 07:34:42.764600 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 07:34:42.764684 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:34:42.764852 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:34:42.764853 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:34:42.765024 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:34:42.765086 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:42.765088 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:42.765837 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:42.765839 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:42.766366 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:42.766368 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:42.766750 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:42.766752 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:42.766833 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:42.767352 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:34:42.767357 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:34:42.768720 (+ 1363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:34:42.770475 (+ 1755us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:34:42.770476 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:34:42.770477 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:34:42.770714 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:34:42.770932 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:34:42.770933 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:34:42.770934 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:34:42.771002 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:34:42.771243 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:34:42.771244 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:34:42.771288 (+ 44us) client_negotiation.cc:769] Sending connection context
0925 07:34:42.771374 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 07:34:42.771473 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260925 07:34:42.772049 32167 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.66.60" port: 34759 } has no permanent_uuid. Determining permanent_uuid...
I20260925 07:34:42.772341 32169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:42.772264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.60:34759 (local address 127.0.0.1:59126)
0925 07:34:42.772299 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0925 07:34:42.772324 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.60:34759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260925 07:34:42.772624 32169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:42.772535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.60:34759 (local address 127.0.0.1:59128)
0925 07:34:42.772581 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0925 07:34:42.772604 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.60:34759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260925 07:34:42.772714 32167 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.66.60:34759: Network error: Client connection negotiation failed: client connection to 127.9.66.60:34759: connect: Connection refused (error 111)
I20260925 07:34:42.803843 32167 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.66.60" port: 34759 } attempt: 1
I20260925 07:34:42.804308 32169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:42.804183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.60:34759 (local address 127.0.0.1:59140)
0925 07:34:42.804243 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0925 07:34:42.804282 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.60:34759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260925 07:34:42.804646 32169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:42.804554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.60:34759 (local address 127.0.0.1:59142)
0925 07:34:42.804598 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0925 07:34:42.804621 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.60:34759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260925 07:34:42.804785 32167 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.66.60:34759: Network error: Client connection negotiation failed: client connection to 127.9.66.60:34759: connect: Connection refused (error 111)
W20260925 07:34:42.805266 32249 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 07:34:42.805487 32249 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 07:34:42.805536 32249 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 07:34:42.805598 32249 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 07:34:42.806687 32249 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 07:34:42.806749 32249 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 07:34:42.806773 32249 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 07:34:42.806802 32249 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 07:34:42.806828 32249 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 07:34:42.808272 32249 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.66.20:42415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790321392278483-9480-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790321392278483-9480-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.9.66.62:41807,127.9.66.61:45751,127.9.66.60:34759
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790321392278483-9480-0/krb5kdc/kudu_127.9.66.60.keytab
--principal=kudu/127.9.66.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.66.60:34759
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790321392278483-9480-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.9.66.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32249
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790321392278483-9480-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 06:22:12 UTC on ccycloud.quasar-vmwxne.root.comops.site
build id 4671942
I20260925 07:34:42.808542 32249 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 07:34:42.808753 32249 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 07:34:42.813230 32258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:34:42.813257 32256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 07:34:42.813257 32255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 07:34:42.813537 32249 server_base.cc:1034] running on GCE node
I20260925 07:34:42.813843 32249 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 07:34:42.814123 32249 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 07:34:42.815254 32249 hybrid_clock.cc:648] HybridClock initialized: now 1790321682815240 us; error 33 us; skew 500 ppm
Sep 25 07:34:42 dist-test-slave-57fn krb5kdc[32086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321682, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 07:34:42.817482 32249 init.cc:377] Logged in from keytab as kudu/127.9.66.60@KRBTEST.COM (short username kudu)
I20260925 07:34:42.817981 32249 webserver.cc:460] Webserver started at http://127.9.66.60:43123/ using document root <none> and password file <none>
I20260925 07:34:42.818197 32249 fs_manager.cc:359] Metadata directory not provided
I20260925 07:34:42.818261 32249 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 07:34:42.818388 32249 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 07:34:42.819345 32249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790321392278483-9480-0/minicluster-data/master-2/data/instance:
uuid: "568eca85abd242088db9e887cb34f3f4"
format_stamp: "Formatted at 2026-09-25 07:34:42 on dist-test-slave-57fn"
server_key: "3f3954159352bec1cc938020fc46cde9"
server_key_iv: "a895d953fce2592f146ff56ca2997e15"
server_key_version: "encryptionkey@0"
I20260925 07:34:42.819689 32249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790321392278483-9480-0/minicluster-data/master-2/wal/instance:
uuid: "568eca85abd242088db9e887cb34f3f4"
format_stamp: "Formatted at 2026-09-25 07:34:42 on dist-test-slave-57fn"
server_key: "3f3954159352bec1cc938020fc46cde9"
server_key_iv: "a895d953fce2592f146ff56ca2997e15"
server_key_version: "encryptionkey@0"
I20260925 07:34:42.820966 32249 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 07:34:42.821738 32265 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 07:34:42.821928 32249 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 07:34:42.822021 32249 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790321392278483-9480-0/minicluster-data/master-2/data,/tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790321392278483-9480-0/minicluster-data/master-2/wal
uuid: "568eca85abd242088db9e887cb34f3f4"
format_stamp: "Formatted at 2026-09-25 07:34:42 on dist-test-slave-57fn"
server_key: "3f3954159352bec1cc938020fc46cde9"
server_key_iv: "a895d953fce2592f146ff56ca2997e15"
server_key_version: "encryptionkey@0"
I20260925 07:34:42.822104 32249 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790321392278483-9480-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790321392278483-9480-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790321392278483-9480-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 07:34:42.838754 32242 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.66.60" port: 34759 } attempt: 2
I20260925 07:34:42.839321 32250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:42.839173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.60:34759 (local address 127.0.0.1:59150)
0925 07:34:42.839246 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0925 07:34:42.839294 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.60:34759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260925 07:34:42.839618 32250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:42.839531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.60:34759 (local address 127.0.0.1:59152)
0925 07:34:42.839574 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0925 07:34:42.839602 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.66.60:34759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260925 07:34:42.839690 32242 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.66.60:34759: Network error: Client connection negotiation failed: client connection to 127.9.66.60:34759: connect: Connection refused (error 111)
I20260925 07:34:42.851261 32249 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 07:34:42.852979 32249 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 07:34:42.853134 32249 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 07:34:42.856894 32249 rpc_server.cc:307] RPC server started. Bound to: 127.9.66.60:34759
I20260925 07:34:42.856917 32317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.66.60:34759 every 8 connection(s)
I20260925 07:34:42.857318 32249 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790321392278483-9480-0/minicluster-data/master-2/data/info.pb
I20260925 07:34:42.858326 32318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 07:34:42.859673 9480 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1eZXJS/build/release/bin/kudu as pid 32249
I20260925 07:34:42.859819 9480 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1eZXJS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790321392278483-9480-0/minicluster-data/master-2/wal/instance
I20260925 07:34:42.860054 9480 external_mini_cluster.cc:1428] Setting key 15137e3fb97894ebe6b9aa0ad66ce7c3
I20260925 07:34:42.859879 32318 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.66.62" port: 41807 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 07:34:42 dist-test-slave-57fn krb5kdc[32086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321682, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.60@KRBTEST.COM for kudu/127.9.66.62@KRBTEST.COM
Sep 25 07:34:42 dist-test-slave-57fn krb5kdc[32086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321682, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.66.60@KRBTEST.COM
I20260925 07:34:42.869526 32170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:42.860574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45452 (local address 127.9.66.62:41807)
0925 07:34:42.860621 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:42.860623 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:42.860858 (+ 235us) server_negotiation.cc:407] Connection header received
0925 07:34:42.861478 (+ 620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:42.861480 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:42.861493 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:42.861626 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:34:42.862107 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:42.862632 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:42.863703 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:42.863981 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:42.864060 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:42.866378 (+ 2318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:34:42.866379 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:34:42.866380 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:34:42.866387 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:34:42.867933 (+ 1546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:34:42.868412 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:34:42.868414 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:34:42.868414 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:34:42.868487 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:34:42.868760 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:34:42.868761 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:34:42.868761 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:34:42.868926 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:34:42.869007 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:42.869275 (+ 268us) server_negotiation.cc:299] Negotiation successful
0925 07:34:42.869384 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 07:34:42.869660 32321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:42.860576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.62:41807 (local address 127.0.0.1:45452)
0925 07:34:42.860743 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 07:34:42.860775 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0925 07:34:42.861305 (+ 530us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:34:42.861645 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:34:42.861648 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:34:42.861875 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:34:42.862011 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:42.862015 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:42.862718 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:42.862719 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:42.863612 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:42.863614 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:42.864092 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:42.864093 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:42.864228 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:42.864825 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:34:42.864832 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:34:42.866262 (+ 1430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:34:42.868039 (+ 1777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:34:42.868046 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:34:42.868049 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:34:42.868305 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:34:42.868571 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:34:42.868572 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:34:42.868572 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:34:42.868684 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:34:42.869006 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:34:42.869011 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:34:42.869116 (+ 105us) client_negotiation.cc:769] Sending connection context
0925 07:34:42.869235 (+ 119us) client_negotiation.cc:240] Negotiation successful
0925 07:34:42.869389 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"mutex_wait_us":60,"thread_start_us":76,"threads_started":1}
I20260925 07:34:42.870234 32318 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.66.61" port: 45751 } has no permanent_uuid. Determining permanent_uuid...
I20260925 07:34:42.870524 32322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:42.860766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59156 (local address 127.9.66.60:34759)
0925 07:34:42.860889 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:42.860894 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:42.860922 (+ 28us) server_negotiation.cc:407] Connection header received
0925 07:34:42.861308 (+ 386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:42.861319 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:42.861345 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:42.861443 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:34:42.862241 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:42.863220 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:42.864118 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:42.864523 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:42.864593 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:42.867202 (+ 2609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:34:42.867209 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:34:42.867210 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:34:42.867217 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:34:42.868817 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:34:42.869191 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:34:42.869194 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:34:42.869194 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:34:42.869253 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:34:42.869467 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:34:42.869468 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:34:42.869468 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:34:42.869643 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:34:42.869776 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:42.870297 (+ 521us) server_negotiation.cc:299] Negotiation successful
0925 07:34:42.870393 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":51,"threads_started":1}
Sep 25 07:34:42 dist-test-slave-57fn krb5kdc[32086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321682, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.60@KRBTEST.COM for kudu/127.9.66.61@KRBTEST.COM
Sep 25 07:34:42 dist-test-slave-57fn krb5kdc[32086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321682, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.66.62@KRBTEST.COM
I20260925 07:34:42.877492 32321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:42.870497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.61:45751 (local address 127.0.0.1:43628)
0925 07:34:42.870604 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0925 07:34:42.870614 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 07:34:42.870665 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:34:42.870846 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:34:42.870847 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:34:42.870959 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:34:42.871033 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:42.871034 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:42.871963 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:42.871963 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:42.872445 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:42.872447 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:42.872866 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:42.872867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:42.872939 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:42.873328 (+ 389us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:34:42.873333 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:34:42.874587 (+ 1254us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:34:42.876404 (+ 1817us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:34:42.876405 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:34:42.876405 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:34:42.876581 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:34:42.876822 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:34:42.876823 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:34:42.876824 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:34:42.876895 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:34:42.877165 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:34:42.877166 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:34:42.877209 (+ 43us) client_negotiation.cc:769] Sending connection context
0925 07:34:42.877303 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 07:34:42.877397 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90}
I20260925 07:34:42.877527 32247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:42.870559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43628 (local address 127.9.66.61:45751)
0925 07:34:42.870601 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:42.870603 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:42.870669 (+ 66us) server_negotiation.cc:407] Connection header received
0925 07:34:42.870712 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:42.870713 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:42.870726 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:42.870787 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:34:42.871127 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:42.871883 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:42.872553 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:42.872776 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:42.872873 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:42.874693 (+ 1820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:34:42.874695 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:34:42.874695 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:34:42.874704 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:34:42.876287 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:34:42.876673 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:34:42.876674 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:34:42.876675 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:34:42.876740 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:34:42.876975 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:34:42.876976 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:34:42.876976 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:34:42.877093 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:34:42.877173 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:42.877329 (+ 156us) server_negotiation.cc:299] Negotiation successful
0925 07:34:42.877408 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 07:34:42.878190 32318 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.66.60" port: 34759 } has no permanent_uuid. Determining permanent_uuid...
I20260925 07:34:42.878453 32170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:42.871284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45468 (local address 127.9.66.62:41807)
0925 07:34:42.871316 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:42.871318 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:42.871363 (+ 45us) server_negotiation.cc:407] Connection header received
0925 07:34:42.871426 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:42.871427 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:42.871435 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:42.871475 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:34:42.872016 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:42.872451 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:42.873124 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:42.873389 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:42.873470 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:42.875898 (+ 2428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:34:42.875899 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:34:42.875900 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:34:42.875906 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:34:42.877262 (+ 1356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:34:42.877690 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:34:42.877691 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:34:42.877691 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:34:42.877744 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:34:42.877949 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:34:42.877950 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:34:42.877950 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:34:42.878056 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:34:42.878229 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:42.878264 (+ 35us) server_negotiation.cc:299] Negotiation successful
0925 07:34:42.878329 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 07:34:42.879895 32167 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.66.60" port: 34759 } attempt: 2
Sep 25 07:34:42 dist-test-slave-57fn krb5kdc[32086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321682, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.60@KRBTEST.COM for kudu/127.9.66.60@KRBTEST.COM
Sep 25 07:34:42 dist-test-slave-57fn krb5kdc[32086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321682, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.66.61@KRBTEST.COM
Sep 25 07:34:42 dist-test-slave-57fn krb5kdc[32086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321682, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.62@KRBTEST.COM for kudu/127.9.66.60@KRBTEST.COM
I20260925 07:34:42.885995 32321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:42.878503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.60:34759 (local address 127.0.0.1:59160)
0925 07:34:42.878534 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0925 07:34:42.878544 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 07:34:42.878585 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:34:42.878851 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:34:42.878852 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:34:42.878996 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:34:42.879024 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:42.879025 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:42.879888 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:42.879889 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:42.880633 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:42.880635 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:42.881034 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:42.881035 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:42.881084 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:42.881447 (+ 363us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:34:42.881451 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:34:42.882820 (+ 1369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:34:42.884446 (+ 1626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:34:42.884447 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:34:42.884447 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:34:42.884905 (+ 458us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:34:42.885108 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:34:42.885109 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:34:42.885109 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:34:42.885407 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:34:42.885692 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:34:42.885693 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:34:42.885728 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 07:34:42.885792 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 07:34:42.885883 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16}
I20260925 07:34:42.886089 32322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:42.878608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59160 (local address 127.9.66.60:34759)
0925 07:34:42.878725 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:42.878731 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:42.878738 (+ 7us) server_negotiation.cc:407] Connection header received
0925 07:34:42.878773 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:42.878774 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:42.878785 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:42.878842 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:34:42.879112 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:42.879813 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:42.880726 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:42.880947 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:42.881034 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:42.882901 (+ 1867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:34:42.882902 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:34:42.882903 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:34:42.882910 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:34:42.884358 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:34:42.884992 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:34:42.884993 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:34:42.884993 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:34:42.885045 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:34:42.885491 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:34:42.885492 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:34:42.885493 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:34:42.885629 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:34:42.885700 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:42.885819 (+ 119us) server_negotiation.cc:299] Negotiation successful
0925 07:34:42.885970 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":50,"server-negotiator.queue_time_us":100}
I20260925 07:34:42.888067 32318 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 568eca85abd242088db9e887cb34f3f4: Bootstrap starting.
I20260925 07:34:42.888463 32169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:42.880115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.60:34759 (local address 127.0.0.1:59170)
0925 07:34:42.880298 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0925 07:34:42.880309 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 07:34:42.880356 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:34:42.880517 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:34:42.880518 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:34:42.880664 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:34:42.880697 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:42.880698 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:42.881654 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:42.881655 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:42.882576 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:42.882579 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:42.883044 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:42.883046 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:42.883100 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:42.884061 (+ 961us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:34:42.884066 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:34:42.885280 (+ 1214us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:34:42.887569 (+ 2289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:34:42.887570 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:34:42.887570 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:34:42.887722 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:34:42.887923 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:34:42.887924 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:34:42.887924 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:34:42.887972 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:34:42.888232 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:34:42.888237 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:34:42.888270 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 07:34:42.888322 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 07:34:42.888386 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167}
I20260925 07:34:42.888561 32323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:42.880161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59170 (local address 127.9.66.60:34759)
0925 07:34:42.880309 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:42.880311 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:42.880342 (+ 31us) server_negotiation.cc:407] Connection header received
0925 07:34:42.880410 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:42.880410 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:42.880419 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:42.880767 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:34:42.880814 (+ 47us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:42.881561 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:42.882661 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:42.882972 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:42.883043 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:42.885768 (+ 2725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:34:42.885769 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:34:42.885770 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:34:42.885776 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:34:42.887495 (+ 1719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:34:42.887797 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:34:42.887798 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:34:42.887798 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:34:42.887856 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:34:42.888027 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:34:42.888028 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:34:42.888028 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:34:42.888172 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:34:42.888238 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:42.888344 (+ 106us) server_negotiation.cc:299] Negotiation successful
0925 07:34:42.888437 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":104,"threads_started":1}
I20260925 07:34:42.888798 32318 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 568eca85abd242088db9e887cb34f3f4: Neither blocks nor log segments found. Creating new log.
I20260925 07:34:42.889125 32318 log.cc:824] T 00000000000000000000000000000000 P 568eca85abd242088db9e887cb34f3f4: Log is configured to *not* fsync() on all Append() calls
I20260925 07:34:42.890005 32318 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 568eca85abd242088db9e887cb34f3f4: No bootstrap required, opened a new log
I20260925 07:34:42.890434 32167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9618ba2b332490ebc7f55e7bf191dce: Bootstrap starting.
I20260925 07:34:42.891094 32167 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c9618ba2b332490ebc7f55e7bf191dce: Neither blocks nor log segments found. Creating new log.
I20260925 07:34:42.891407 32167 log.cc:824] T 00000000000000000000000000000000 P c9618ba2b332490ebc7f55e7bf191dce: Log is configured to *not* fsync() on all Append() calls
I20260925 07:34:42.892128 32167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9618ba2b332490ebc7f55e7bf191dce: No bootstrap required, opened a new log
I20260925 07:34:42.892155 32318 raft_consensus.cc:348] T 00000000000000000000000000000000 P 568eca85abd242088db9e887cb34f3f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9618ba2b332490ebc7f55e7bf191dce" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 41807 } } peers { permanent_uuid: "87b96a210f44439e9b395de6d3eb236d" member_type: VOTER last_known_addr { host: "127.9.66.61" port: 45751 } } peers { permanent_uuid: "568eca85abd242088db9e887cb34f3f4" member_type: VOTER last_known_addr { host: "127.9.66.60" port: 34759 } }
I20260925 07:34:42.892298 32318 raft_consensus.cc:729] T 00000000000000000000000000000000 P 568eca85abd242088db9e887cb34f3f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 568eca85abd242088db9e887cb34f3f4, State: Initialized, Role: FOLLOWER
I20260925 07:34:42.892382 32318 consensus_queue.cc:260] T 00000000000000000000000000000000 P 568eca85abd242088db9e887cb34f3f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9618ba2b332490ebc7f55e7bf191dce" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 41807 } } peers { permanent_uuid: "87b96a210f44439e9b395de6d3eb236d" member_type: VOTER last_known_addr { host: "127.9.66.61" port: 45751 } } peers { permanent_uuid: "568eca85abd242088db9e887cb34f3f4" member_type: VOTER last_known_addr { host: "127.9.66.60" port: 34759 } }
I20260925 07:34:42.892663 32318 sys_catalog.cc:564] T 00000000000000000000000000000000 P 568eca85abd242088db9e887cb34f3f4 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:34:42.892632 32326 sys_catalog.cc:455] T 00000000000000000000000000000000 P 568eca85abd242088db9e887cb34f3f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9618ba2b332490ebc7f55e7bf191dce" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 41807 } } peers { permanent_uuid: "87b96a210f44439e9b395de6d3eb236d" member_type: VOTER last_known_addr { host: "127.9.66.61" port: 45751 } } peers { permanent_uuid: "568eca85abd242088db9e887cb34f3f4" member_type: VOTER last_known_addr { host: "127.9.66.60" port: 34759 } } }
I20260925 07:34:42.892730 32326 sys_catalog.cc:458] T 00000000000000000000000000000000 P 568eca85abd242088db9e887cb34f3f4 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 07:34:42.893853 32247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:42.879084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43640 (local address 127.9.66.61:45751)
0925 07:34:42.879126 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:42.879128 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:42.879136 (+ 8us) server_negotiation.cc:407] Connection header received
0925 07:34:42.879156 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:42.879156 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:42.879165 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:42.879221 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:34:42.879736 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:42.880408 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:42.881610 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:42.881937 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:42.882007 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:42.884482 (+ 2475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:34:42.884483 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:34:42.884484 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:34:42.884490 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:34:42.886165 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:34:42.886590 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:34:42.886592 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:34:42.886592 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:34:42.886646 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:34:42.889366 (+ 2720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:34:42.889367 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:34:42.889368 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:34:42.889485 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:34:42.893616 (+ 4131us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:42.893668 (+ 52us) server_negotiation.cc:299] Negotiation successful
0925 07:34:42.893750 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 07:34:42.894384 32167 raft_consensus.cc:348] T 00000000000000000000000000000000 P c9618ba2b332490ebc7f55e7bf191dce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9618ba2b332490ebc7f55e7bf191dce" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 41807 } } peers { permanent_uuid: "87b96a210f44439e9b395de6d3eb236d" member_type: VOTER last_known_addr { host: "127.9.66.61" port: 45751 } } peers { permanent_uuid: "568eca85abd242088db9e887cb34f3f4" member_type: VOTER last_known_addr { host: "127.9.66.60" port: 34759 } }
I20260925 07:34:42.894537 32167 raft_consensus.cc:729] T 00000000000000000000000000000000 P c9618ba2b332490ebc7f55e7bf191dce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9618ba2b332490ebc7f55e7bf191dce, State: Initialized, Role: FOLLOWER
I20260925 07:34:42.894618 32167 consensus_queue.cc:260] T 00000000000000000000000000000000 P c9618ba2b332490ebc7f55e7bf191dce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9618ba2b332490ebc7f55e7bf191dce" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 41807 } } peers { permanent_uuid: "87b96a210f44439e9b395de6d3eb236d" member_type: VOTER last_known_addr { host: "127.9.66.61" port: 45751 } } peers { permanent_uuid: "568eca85abd242088db9e887cb34f3f4" member_type: VOTER last_known_addr { host: "127.9.66.60" port: 34759 } }
W20260925 07:34:42.894874 32337 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 568eca85abd242088db9e887cb34f3f4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260925 07:34:42.894920 32167 sys_catalog.cc:564] T 00000000000000000000000000000000 P c9618ba2b332490ebc7f55e7bf191dce [sys.catalog]: configured and running, proceeding with master startup.
W20260925 07:34:42.894960 32337 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 07:34:42.895155 9480 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 07:34:42.896092 32141 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "c9618ba2b332490ebc7f55e7bf191dce"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45468
I20260925 07:34:42.896205 32141 raft_consensus.cc:482] T 00000000000000000000000000000000 P c9618ba2b332490ebc7f55e7bf191dce [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 07:34:42.896245 32141 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c9618ba2b332490ebc7f55e7bf191dce [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:34:42.897009 32141 raft_consensus.cc:504] T 00000000000000000000000000000000 P c9618ba2b332490ebc7f55e7bf191dce [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9618ba2b332490ebc7f55e7bf191dce" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 41807 } } peers { permanent_uuid: "87b96a210f44439e9b395de6d3eb236d" member_type: VOTER last_known_addr { host: "127.9.66.61" port: 45751 } } peers { permanent_uuid: "568eca85abd242088db9e887cb34f3f4" member_type: VOTER last_known_addr { host: "127.9.66.60" port: 34759 } }
I20260925 07:34:42.897239 32141 leader_election.cc:288] T 00000000000000000000000000000000 P c9618ba2b332490ebc7f55e7bf191dce [CANDIDATE]: Term 1 election: Requested vote from peers 87b96a210f44439e9b395de6d3eb236d (127.9.66.61:45751), 568eca85abd242088db9e887cb34f3f4 (127.9.66.60:34759)
I20260925 07:34:42.897711 32292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c9618ba2b332490ebc7f55e7bf191dce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "568eca85abd242088db9e887cb34f3f4"
W20260925 07:34:42.897759 32350 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c9618ba2b332490ebc7f55e7bf191dce: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 07:34:42.897804 32350 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 07:34:42.897805 32292 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 568eca85abd242088db9e887cb34f3f4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:34:42.898027 32339 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9618ba2b332490ebc7f55e7bf191dce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9618ba2b332490ebc7f55e7bf191dce" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 41807 } } peers { permanent_uuid: "87b96a210f44439e9b395de6d3eb236d" member_type: VOTER last_known_addr { host: "127.9.66.61" port: 45751 } } peers { permanent_uuid: "568eca85abd242088db9e887cb34f3f4" member_type: VOTER last_known_addr { host: "127.9.66.60" port: 34759 } } }
I20260925 07:34:42.898152 32339 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9618ba2b332490ebc7f55e7bf191dce [sys.catalog]: This master's current role is: FOLLOWER
I20260925 07:34:42.898569 32292 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 568eca85abd242088db9e887cb34f3f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9618ba2b332490ebc7f55e7bf191dce in term 1.
I20260925 07:34:42.899746 32117 leader_election.cc:302] T 00000000000000000000000000000000 P c9618ba2b332490ebc7f55e7bf191dce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 568eca85abd242088db9e887cb34f3f4, c9618ba2b332490ebc7f55e7bf191dce; no voters:
I20260925 07:34:42.899863 32339 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c9618ba2b332490ebc7f55e7bf191dce [term 1 FOLLOWER]: Leader election won for term 1
I20260925 07:34:42.900002 32339 raft_consensus.cc:686] T 00000000000000000000000000000000 P c9618ba2b332490ebc7f55e7bf191dce [term 1 LEADER]: Becoming Leader. State: Replica: c9618ba2b332490ebc7f55e7bf191dce, State: Running, Role: LEADER
I20260925 07:34:42.900076 32339 consensus_queue.cc:237] T 00000000000000000000000000000000 P c9618ba2b332490ebc7f55e7bf191dce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9618ba2b332490ebc7f55e7bf191dce" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 41807 } } peers { permanent_uuid: "87b96a210f44439e9b395de6d3eb236d" member_type: VOTER last_known_addr { host: "127.9.66.61" port: 45751 } } peers { permanent_uuid: "568eca85abd242088db9e887cb34f3f4" member_type: VOTER last_known_addr { host: "127.9.66.60" port: 34759 } }
I20260925 07:34:42.898202 32099 master_runner.cc:186] Error getting master registration for 127.9.66.61:45751: OK, instance_id { permanent_uuid: "87b96a210f44439e9b395de6d3eb236d" instance_seqno: 1790321682695965 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 07:34:42.900456 32099 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 07:34:42.900437 32339 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9618ba2b332490ebc7f55e7bf191dce [sys.catalog]: SysCatalogTable state changed. Reason: New leader c9618ba2b332490ebc7f55e7bf191dce. Latest consensus state: current_term: 1 leader_uuid: "c9618ba2b332490ebc7f55e7bf191dce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9618ba2b332490ebc7f55e7bf191dce" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 41807 } } peers { permanent_uuid: "87b96a210f44439e9b395de6d3eb236d" member_type: VOTER last_known_addr { host: "127.9.66.61" port: 45751 } } peers { permanent_uuid: "568eca85abd242088db9e887cb34f3f4" member_type: VOTER last_known_addr { host: "127.9.66.60" port: 34759 } } }
I20260925 07:34:42.900581 32339 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9618ba2b332490ebc7f55e7bf191dce [sys.catalog]: This master's current role is: LEADER
I20260925 07:34:42.901822 32249 master_runner.cc:186] Error getting master registration for 127.9.66.61:45751: OK, instance_id { permanent_uuid: "87b96a210f44439e9b395de6d3eb236d" instance_seqno: 1790321682695965 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 07:34:42.906075 32249 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 07:34:42.906628 32358 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 07:34:42.906831 32358 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 07:34:42.913303 32292 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 568eca85abd242088db9e887cb34f3f4 [term 1 FOLLOWER]: Refusing update from remote peer c9618ba2b332490ebc7f55e7bf191dce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:34:42.913758 32356 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c9618ba2b332490ebc7f55e7bf191dce [LEADER]: Connected to new peer: Peer: permanent_uuid: "568eca85abd242088db9e887cb34f3f4" member_type: VOTER last_known_addr { host: "127.9.66.60" port: 34759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:34:42.915006 32326 sys_catalog.cc:455] T 00000000000000000000000000000000 P 568eca85abd242088db9e887cb34f3f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c9618ba2b332490ebc7f55e7bf191dce. Latest consensus state: current_term: 1 leader_uuid: "c9618ba2b332490ebc7f55e7bf191dce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9618ba2b332490ebc7f55e7bf191dce" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 41807 } } peers { permanent_uuid: "87b96a210f44439e9b395de6d3eb236d" member_type: VOTER last_known_addr { host: "127.9.66.61" port: 45751 } } peers { permanent_uuid: "568eca85abd242088db9e887cb34f3f4" member_type: VOTER last_known_addr { host: "127.9.66.60" port: 34759 } } }
I20260925 07:34:42.915094 32326 sys_catalog.cc:458] T 00000000000000000000000000000000 P 568eca85abd242088db9e887cb34f3f4 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 07:34:42.917790 32247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:42.905826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43654 (local address 127.9.66.61:45751)
0925 07:34:42.905878 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:42.905880 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:42.907624 (+ 1744us) server_negotiation.cc:407] Connection header received
0925 07:34:42.907719 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:42.907720 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:42.907730 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:42.907759 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:34:42.908500 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:42.909226 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:42.910460 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:42.910775 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:42.910821 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:42.913471 (+ 2650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:34:42.913472 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:34:42.913473 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:34:42.913479 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:34:42.915099 (+ 1620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:34:42.916376 (+ 1277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:34:42.916377 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:34:42.916377 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:34:42.916438 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:34:42.916779 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:34:42.916779 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:34:42.916780 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:34:42.917219 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:34:42.917292 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:42.917622 (+ 330us) server_negotiation.cc:299] Negotiation successful
0925 07:34:42.917698 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 07:34:42.920732 32170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:42.906040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45470 (local address 127.9.66.62:41807)
0925 07:34:42.906080 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:42.906082 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:42.906401 (+ 319us) server_negotiation.cc:407] Connection header received
0925 07:34:42.906558 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:42.906559 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:42.906580 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:42.906608 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:34:42.907908 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:42.908695 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:42.910273 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:42.910566 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:42.910604 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:42.913614 (+ 3010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:34:42.913615 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:34:42.913615 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:34:42.913621 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:34:42.915883 (+ 2262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:34:42.919701 (+ 3818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:34:42.919702 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:34:42.919702 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:34:42.919770 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:34:42.920066 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:34:42.920067 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:34:42.920067 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:34:42.920201 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:34:42.920288 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:42.920572 (+ 284us) server_negotiation.cc:299] Negotiation successful
0925 07:34:42.920646 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 07:34:42.922091 32323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:42.906386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59182 (local address 127.9.66.60:34759)
0925 07:34:42.906424 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:42.906425 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:42.910358 (+ 3933us) server_negotiation.cc:407] Connection header received
0925 07:34:42.910426 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:42.910427 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:42.910439 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:42.910486 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:34:42.912449 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:42.913129 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:42.916125 (+ 2996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:42.916447 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:42.916483 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:42.918484 (+ 2001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:34:42.918485 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:34:42.918486 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:34:42.918504 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:34:42.920086 (+ 1582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:34:42.920876 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:34:42.920877 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:34:42.920878 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:34:42.920947 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:34:42.921213 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:34:42.921214 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:34:42.921214 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:34:42.921334 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:34:42.921426 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:42.921927 (+ 501us) server_negotiation.cc:299] Negotiation successful
0925 07:34:42.922008 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 07:34:42.923568 32339 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9618ba2b332490ebc7f55e7bf191dce [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c9618ba2b332490ebc7f55e7bf191dce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9618ba2b332490ebc7f55e7bf191dce" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 41807 } } peers { permanent_uuid: "87b96a210f44439e9b395de6d3eb236d" member_type: VOTER last_known_addr { host: "127.9.66.61" port: 45751 } } peers { permanent_uuid: "568eca85abd242088db9e887cb34f3f4" member_type: VOTER last_known_addr { host: "127.9.66.60" port: 34759 } } }
I20260925 07:34:42.923642 32339 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9618ba2b332490ebc7f55e7bf191dce [sys.catalog]: This master's current role is: LEADER
I20260925 07:34:42.923676 32326 sys_catalog.cc:455] T 00000000000000000000000000000000 P 568eca85abd242088db9e887cb34f3f4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c9618ba2b332490ebc7f55e7bf191dce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9618ba2b332490ebc7f55e7bf191dce" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 41807 } } peers { permanent_uuid: "87b96a210f44439e9b395de6d3eb236d" member_type: VOTER last_known_addr { host: "127.9.66.61" port: 45751 } } peers { permanent_uuid: "568eca85abd242088db9e887cb34f3f4" member_type: VOTER last_known_addr { host: "127.9.66.60" port: 34759 } } }
I20260925 07:34:42.923754 32326 sys_catalog.cc:458] T 00000000000000000000000000000000 P 568eca85abd242088db9e887cb34f3f4 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 07:34:42.923776 32362 mvcc.cc:205] Tried to move back new op lower bound from 7333157613207572480 to 7333157613159297024. Current Snapshot: MvccSnapshot[applied={T|T < 7333157613207572480}]
I20260925 07:34:42.924485 32363 mvcc.cc:205] Tried to move back new op lower bound from 7333157613207572480 to 7333157613159297024. Current Snapshot: MvccSnapshot[applied={T|T < 7333157613207572480}]
I20260925 07:34:42.929838 32358 catalog_manager.cc:1347] Generated new cluster ID: acc0eb2dbd4c4d5ea3ac30531ab9e2ed
I20260925 07:34:42.929893 32358 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 07:34:42.930809 32242 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.66.60" port: 34759 } attempt: 3
Sep 25 07:34:42 dist-test-slave-57fn krb5kdc[32086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790321682, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.66.61@KRBTEST.COM for kudu/127.9.66.60@KRBTEST.COM
I20260925 07:34:42.938293 32250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:42.931036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.66.60:34759 (local address 127.0.0.1:59188)
0925 07:34:42.931082 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0925 07:34:42.931092 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 07:34:42.931139 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 07:34:42.931292 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 07:34:42.931293 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 07:34:42.931475 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0925 07:34:42.931550 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:42.931553 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:42.932470 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:42.932471 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:42.933320 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 07:34:42.933322 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 07:34:42.933892 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 07:34:42.933892 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 07:34:42.933967 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:42.934356 (+ 389us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 07:34:42.934360 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 07:34:42.935672 (+ 1312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 07:34:42.937196 (+ 1524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:34:42.937197 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:34:42.937197 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:34:42.937377 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:34:42.937563 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 07:34:42.937563 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 07:34:42.937564 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 07:34:42.937648 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 07:34:42.937986 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 07:34:42.937987 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 07:34:42.938044 (+ 57us) client_negotiation.cc:769] Sending connection context
0925 07:34:42.938105 (+ 61us) client_negotiation.cc:240] Negotiation successful
0925 07:34:42.938197 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260925 07:34:42.938894 32323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:42.931081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59188 (local address 127.9.66.60:34759)
0925 07:34:42.931115 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:42.931117 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:42.931126 (+ 9us) server_negotiation.cc:407] Connection header received
0925 07:34:42.931187 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:42.931188 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:42.931209 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:42.931254 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:34:42.931648 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:42.932367 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:42.933433 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:42.933773 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:42.933831 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:42.935749 (+ 1918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 07:34:42.935750 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 07:34:42.935750 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 07:34:42.935756 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 07:34:42.937121 (+ 1365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:34:42.937446 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:34:42.937446 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:34:42.937447 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:34:42.937486 (+ 39us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 07:34:42.937735 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 07:34:42.937736 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 07:34:42.937736 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 07:34:42.937882 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 07:34:42.937944 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:42.938742 (+ 798us) server_negotiation.cc:299] Negotiation successful
0925 07:34:42.938812 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 07:34:42.940243 32242 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87b96a210f44439e9b395de6d3eb236d: Bootstrap starting.
I20260925 07:34:42.940925 32242 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 87b96a210f44439e9b395de6d3eb236d: Neither blocks nor log segments found. Creating new log.
I20260925 07:34:42.941232 32242 log.cc:824] T 00000000000000000000000000000000 P 87b96a210f44439e9b395de6d3eb236d: Log is configured to *not* fsync() on all Append() calls
I20260925 07:34:42.943118 32242 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87b96a210f44439e9b395de6d3eb236d: No bootstrap required, opened a new log
I20260925 07:34:42.944996 32242 raft_consensus.cc:348] T 00000000000000000000000000000000 P 87b96a210f44439e9b395de6d3eb236d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9618ba2b332490ebc7f55e7bf191dce" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 41807 } } peers { permanent_uuid: "87b96a210f44439e9b395de6d3eb236d" member_type: VOTER last_known_addr { host: "127.9.66.61" port: 45751 } } peers { permanent_uuid: "568eca85abd242088db9e887cb34f3f4" member_type: VOTER last_known_addr { host: "127.9.66.60" port: 34759 } }
I20260925 07:34:42.945130 32242 raft_consensus.cc:729] T 00000000000000000000000000000000 P 87b96a210f44439e9b395de6d3eb236d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87b96a210f44439e9b395de6d3eb236d, State: Initialized, Role: FOLLOWER
I20260925 07:34:42.945192 32242 consensus_queue.cc:260] T 00000000000000000000000000000000 P 87b96a210f44439e9b395de6d3eb236d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9618ba2b332490ebc7f55e7bf191dce" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 41807 } } peers { permanent_uuid: "87b96a210f44439e9b395de6d3eb236d" member_type: VOTER last_known_addr { host: "127.9.66.61" port: 45751 } } peers { permanent_uuid: "568eca85abd242088db9e887cb34f3f4" member_type: VOTER last_known_addr { host: "127.9.66.60" port: 34759 } }
I20260925 07:34:42.945473 32242 sys_catalog.cc:564] T 00000000000000000000000000000000 P 87b96a210f44439e9b395de6d3eb236d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 07:34:42.946887 32216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c9618ba2b332490ebc7f55e7bf191dce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "87b96a210f44439e9b395de6d3eb236d"
I20260925 07:34:42.946826 32215 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 87b96a210f44439e9b395de6d3eb236d [term 0 FOLLOWER]: Advancing to term 1
I20260925 07:34:42.948118 32215 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 87b96a210f44439e9b395de6d3eb236d [term 1 FOLLOWER]: Refusing update from remote peer c9618ba2b332490ebc7f55e7bf191dce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 07:34:42.948406 32356 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c9618ba2b332490ebc7f55e7bf191dce [LEADER]: Connected to new peer: Peer: permanent_uuid: "87b96a210f44439e9b395de6d3eb236d" member_type: VOTER last_known_addr { host: "127.9.66.61" port: 45751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 07:34:42.948961 32216 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 87b96a210f44439e9b395de6d3eb236d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9618ba2b332490ebc7f55e7bf191dce in term 1.
I20260925 07:34:42.950964 32368 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87b96a210f44439e9b395de6d3eb236d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c9618ba2b332490ebc7f55e7bf191dce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9618ba2b332490ebc7f55e7bf191dce" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 41807 } } peers { permanent_uuid: "87b96a210f44439e9b395de6d3eb236d" member_type: VOTER last_known_addr { host: "127.9.66.61" port: 45751 } } peers { permanent_uuid: "568eca85abd242088db9e887cb34f3f4" member_type: VOTER last_known_addr { host: "127.9.66.60" port: 34759 } } }
I20260925 07:34:42.951057 32368 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87b96a210f44439e9b395de6d3eb236d [sys.catalog]: This master's current role is: FOLLOWER
I20260925 07:34:42.950953 32381 mvcc.cc:205] Tried to move back new op lower bound from 7333157613207572480 to 7333157613159297024. Current Snapshot: MvccSnapshot[applied={T|T < 7333157613207572480}]
I20260925 07:34:42.951215 32380 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87b96a210f44439e9b395de6d3eb236d [sys.catalog]: SysCatalogTable state changed. Reason: New leader c9618ba2b332490ebc7f55e7bf191dce. Latest consensus state: current_term: 1 leader_uuid: "c9618ba2b332490ebc7f55e7bf191dce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9618ba2b332490ebc7f55e7bf191dce" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 41807 } } peers { permanent_uuid: "87b96a210f44439e9b395de6d3eb236d" member_type: VOTER last_known_addr { host: "127.9.66.61" port: 45751 } } peers { permanent_uuid: "568eca85abd242088db9e887cb34f3f4" member_type: VOTER last_known_addr { host: "127.9.66.60" port: 34759 } } }
I20260925 07:34:42.951336 32380 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87b96a210f44439e9b395de6d3eb236d [sys.catalog]: This master's current role is: FOLLOWER
I20260925 07:34:42.951426 32368 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87b96a210f44439e9b395de6d3eb236d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c9618ba2b332490ebc7f55e7bf191dce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9618ba2b332490ebc7f55e7bf191dce" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 41807 } } peers { permanent_uuid: "87b96a210f44439e9b395de6d3eb236d" member_type: VOTER last_known_addr { host: "127.9.66.61" port: 45751 } } peers { permanent_uuid: "568eca85abd242088db9e887cb34f3f4" member_type: VOTER last_known_addr { host: "127.9.66.60" port: 34759 } } }
I20260925 07:34:42.951488 32368 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87b96a210f44439e9b395de6d3eb236d [sys.catalog]: This master's current role is: FOLLOWER
W20260925 07:34:42.951835 32379 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 87b96a210f44439e9b395de6d3eb236d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 07:34:42.951884 32379 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 07:34:42.962909 32339 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9618ba2b332490ebc7f55e7bf191dce [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c9618ba2b332490ebc7f55e7bf191dce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9618ba2b332490ebc7f55e7bf191dce" member_type: VOTER last_known_addr { host: "127.9.66.62" port: 41807 } } peers { permanent_uuid: "87b96a210f44439e9b395de6d3eb236d" member_type: VOTER last_known_addr { host: "127.9.66.61" port: 45751 } } peers { permanent_uuid: "568eca85abd242088db9e887cb34f3f4" member_type: VOTER last_known_addr { host: "127.9.66.60" port: 34759 } } }
I20260925 07:34:42.963013 32339 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9618ba2b332490ebc7f55e7bf191dce [sys.catalog]: This master's current role is: LEADER
I20260925 07:34:42.963762 32358 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 07:34:42.964471 32358 catalog_manager.cc:1504] Loading token signing keys...
I20260925 07:34:42.976246 32358 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c9618ba2b332490ebc7f55e7bf191dce: Generated new TSK 0
I20260925 07:34:42.976437 32358 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 07:34:43.000183 32170 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:34:42.998462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45472 (local address 127.9.66.62:41807)
0925 07:34:42.998504 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:42.998506 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:42.999505 (+ 999us) server_negotiation.cc:407] Connection header received
0925 07:34:42.999615 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:42.999616 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:42.999627 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:42.999660 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:34:43.000119 (+ 459us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45472: BlockingRecv error: recv got EOF from 127.0.0.1:45472 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260925 07:34:43.000919 32247 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:34:42.998912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43668 (local address 127.9.66.61:45751)
0925 07:34:42.998960 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:42.998961 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:43.000416 (+ 1455us) server_negotiation.cc:407] Connection header received
0925 07:34:43.000513 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:43.000514 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:43.000525 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:43.000553 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:34:43.000847 (+ 294us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43668: BlockingRecv error: recv got EOF from 127.0.0.1:43668 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
W20260925 07:34:43.001657 32323 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:34:42.999229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59198 (local address 127.9.66.60:34759)
0925 07:34:42.999267 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:42.999269 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:43.001154 (+ 1885us) server_negotiation.cc:407] Connection header received
0925 07:34:43.001272 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:43.001273 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:43.001282 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:43.001310 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:34:43.001601 (+ 291us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59198: BlockingRecv error: recv got EOF from 127.0.0.1:59198 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260925 07:34:43.003734 32247 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:34:43.003061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43678 (local address 127.9.66.61:45751)
0925 07:34:43.003103 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:43.003104 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:43.003284 (+ 180us) server_negotiation.cc:407] Connection header received
0925 07:34:43.003381 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:43.003382 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:43.003398 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:43.003424 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:34:43.003695 (+ 271us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43678: BlockingRecv error: recv got EOF from 127.0.0.1:43678 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260925 07:34:43.006670 32247 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:34:43.005945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43686 (local address 127.9.66.61:45751)
0925 07:34:43.005976 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:43.005978 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:43.006206 (+ 228us) server_negotiation.cc:407] Connection header received
0925 07:34:43.006297 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:43.006298 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:43.006307 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:43.006335 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:34:43.006630 (+ 295us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43686: BlockingRecv error: recv got EOF from 127.0.0.1:43686 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260925 07:34:43.010529 32247 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:34:43.009757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43692 (local address 127.9.66.61:45751)
0925 07:34:43.009788 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:43.009790 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:43.010052 (+ 262us) server_negotiation.cc:407] Connection header received
0925 07:34:43.010164 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:43.010165 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:43.010174 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:43.010203 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:34:43.010491 (+ 288us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43692: BlockingRecv error: recv got EOF from 127.0.0.1:43692 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260925 07:34:43.016222 32247 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:34:43.015478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43694 (local address 127.9.66.61:45751)
0925 07:34:43.015508 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:43.015510 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:43.015724 (+ 214us) server_negotiation.cc:407] Connection header received
0925 07:34:43.015813 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:43.015814 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:43.015823 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:43.015850 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:34:43.016184 (+ 334us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43694: BlockingRecv error: recv got EOF from 127.0.0.1:43694 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260925 07:34:43.026039 32247 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:34:43.025280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43708 (local address 127.9.66.61:45751)
0925 07:34:43.025311 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:43.025313 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:43.025553 (+ 240us) server_negotiation.cc:407] Connection header received
0925 07:34:43.025648 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:43.025649 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:43.025659 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:43.025687 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:34:43.025997 (+ 310us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43708: BlockingRecv error: recv got EOF from 127.0.0.1:43708 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260925 07:34:43.044023 32247 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:34:43.043160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43718 (local address 127.9.66.61:45751)
0925 07:34:43.043195 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:43.043197 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:43.043452 (+ 255us) server_negotiation.cc:407] Connection header received
0925 07:34:43.043550 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:43.043551 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:43.043561 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:43.043589 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:34:43.043961 (+ 372us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43718: BlockingRecv error: recv got EOF from 127.0.0.1:43718 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260925 07:34:43.078096 32247 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:34:43.077264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43732 (local address 127.9.66.61:45751)
0925 07:34:43.077309 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:43.077311 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:43.077447 (+ 136us) server_negotiation.cc:407] Connection header received
0925 07:34:43.077568 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:43.077569 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:43.077583 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:43.077683 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:34:43.078034 (+ 351us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43732: BlockingRecv error: recv got EOF from 127.0.0.1:43732 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260925 07:34:43.144037 32247 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:34:43.143253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43748 (local address 127.9.66.61:45751)
0925 07:34:43.143346 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:43.143349 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:43.143432 (+ 83us) server_negotiation.cc:407] Connection header received
0925 07:34:43.143509 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:43.143510 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:43.143521 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:43.143571 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:34:43.143969 (+ 398us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43748: BlockingRecv error: recv got EOF from 127.0.0.1:43748 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260925 07:34:43.273892 32247 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:34:43.273167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43754 (local address 127.9.66.61:45751)
0925 07:34:43.273249 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:43.273251 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:43.273295 (+ 44us) server_negotiation.cc:407] Connection header received
0925 07:34:43.273345 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:43.273346 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:43.273357 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:43.273410 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:34:43.273797 (+ 387us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43754: BlockingRecv error: recv got EOF from 127.0.0.1:43754 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260925 07:34:43.531891 32247 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 07:34:43.531043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48778 (local address 127.9.66.61:45751)
0925 07:34:43.531109 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:43.531112 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:43.531122 (+ 10us) server_negotiation.cc:407] Connection header received
0925 07:34:43.531210 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:43.531211 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:43.531221 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:43.531274 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 07:34:43.531812 (+ 538us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48778: BlockingRecv error: recv got EOF from 127.0.0.1:48778 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 07:34:43.895442 32337 catalog_manager.cc:1259] Loaded cluster ID: acc0eb2dbd4c4d5ea3ac30531ab9e2ed
I20260925 07:34:43.895510 32337 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 568eca85abd242088db9e887cb34f3f4: loading cluster ID for follower catalog manager: success
I20260925 07:34:43.896363 32337 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 568eca85abd242088db9e887cb34f3f4: acquiring CA information for follower catalog manager: success
I20260925 07:34:43.896641 32337 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 568eca85abd242088db9e887cb34f3f4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 07:34:43.952257 32379 catalog_manager.cc:1259] Loaded cluster ID: acc0eb2dbd4c4d5ea3ac30531ab9e2ed
I20260925 07:34:43.952319 32379 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 87b96a210f44439e9b395de6d3eb236d: loading cluster ID for follower catalog manager: success
I20260925 07:34:43.953032 32379 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 87b96a210f44439e9b395de6d3eb236d: acquiring CA information for follower catalog manager: success
I20260925 07:34:43.953254 32379 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 87b96a210f44439e9b395de6d3eb236d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 07:34:44.047542 32450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:44.044874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48788 (local address 127.9.66.61:45751)
0925 07:34:44.045005 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:44.045007 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:44.045017 (+ 10us) server_negotiation.cc:407] Connection header received
0925 07:34:44.045062 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:44.045062 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:44.045113 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:44.045187 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 07:34:44.045366 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:44.045865 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:44.046742 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:44.046952 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:44.047014 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:44.047174 (+ 160us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 07:34:44.047281 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 07:34:44.047341 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:44.047392 (+ 51us) server_negotiation.cc:299] Negotiation successful
0925 07:34:44.047412 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":64,"threads_started":1}
I20260925 07:34:44.051513 32456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:44.049292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40310 (local address 127.9.66.60:34759)
0925 07:34:44.049419 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:44.049421 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:44.049429 (+ 8us) server_negotiation.cc:407] Connection header received
0925 07:34:44.049463 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:44.049463 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:44.049496 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:44.049610 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 07:34:44.049699 (+ 89us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:44.050179 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:44.050854 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:44.051057 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:44.051121 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:44.051243 (+ 122us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 07:34:44.051332 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 07:34:44.051387 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:44.051437 (+ 50us) server_negotiation.cc:299] Negotiation successful
0925 07:34:44.051454 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86,"thread_start_us":64,"threads_started":1}
I20260925 07:34:44.055936 32450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:44.053100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48800 (local address 127.9.66.61:45751)
0925 07:34:44.053253 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:44.053254 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:44.053383 (+ 129us) server_negotiation.cc:407] Connection header received
0925 07:34:44.053409 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:44.053410 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:44.053420 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:44.053468 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 07:34:44.053642 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:44.054225 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:44.055186 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:44.055474 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:44.055509 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:44.055672 (+ 163us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 07:34:44.055705 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 07:34:44.055749 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:44.055842 (+ 93us) server_negotiation.cc:299] Negotiation successful
0925 07:34:44.055871 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134}
I20260925 07:34:44.056308 32456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:44.053276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40320 (local address 127.9.66.60:34759)
0925 07:34:44.053317 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:44.053318 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:44.053699 (+ 381us) server_negotiation.cc:407] Connection header received
0925 07:34:44.053723 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:44.053724 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:44.053734 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:44.053780 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 07:34:44.053920 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:44.054606 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:44.055350 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:44.055654 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:44.055721 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:44.055860 (+ 139us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 07:34:44.055971 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 07:34:44.056034 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:44.056199 (+ 165us) server_negotiation.cc:299] Negotiation successful
0925 07:34:44.056220 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 07:34:44.056414 32463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 07:34:44.053153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40930 (local address 127.9.66.62:41807)
0925 07:34:44.053350 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 07:34:44.053352 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 07:34:44.053360 (+ 8us) server_negotiation.cc:407] Connection header received
0925 07:34:44.053405 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 07:34:44.053406 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 07:34:44.053474 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 07:34:44.053565 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 07:34:44.053684 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:44.054508 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:44.055485 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 07:34:44.055782 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 07:34:44.055840 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 07:34:44.056081 (+ 241us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 07:34:44.056172 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 07:34:44.056231 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 07:34:44.056300 (+ 69us) server_negotiation.cc:299] Negotiation successful
0925 07:34:44.056320 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":60,"threads_started":1}
I20260925 07:34:44.064831 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 32099
I20260925 07:34:44.070245 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 32174
I20260925 07:34:44.075138 9480 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1eZXJS/build/release/bin/kudu with pid 32249
2026-09-25T07:34:44Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2805 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5563 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (291804 ms total)
[ PASSED ] 54 tests.
[ FAILED ] 2 tests, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
[ FAILED ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
2 FAILED TESTS